15 Commits

Author SHA1 Message Date
shoko
5bfec66a34 docs(polymarket-browse): remove Changelog and Credits from SKILL.md
Changelog tracked via git tags instead.
2026-03-27 02:47:09 +00:00
shoko
0902bfafaa docs(polymarket-browse): remove SECURITY.md - overhead for users/agents
Security findings tracked in GitHub issues instead.
2026-03-27 02:44:05 +00:00
shoko
3d7b136cca docs(polymarket-browse): update SKILL.md for v0.0.3 release
- Bump version to 0.0.3
- Update changelog with all security fixes and features
- Add TokenBucket rate limiting to Rate Limiting section
- Add URL encoding note
2026-03-27 02:40:51 +00:00
shoko
b9ad8ac41b Merge remote-tracking branch 'origin/0.0.3-draft' into docs/11-12-skill-updates 2026-03-27 02:36:54 +00:00
shoko
efad771e85 Merge branch 'pr-38' into 0.0.3-draft 2026-03-27 02:28:27 +00:00
shoko
a2aea41ae3 Merge branch 'pr-36' (rate limiting) into 0.0.3-draft
Combined MAX_RESPONSE_SIZE dynamic calculation with RateLimiter class.
2026-03-27 02:28:16 +00:00
shoko
54679cac44 Merge branch 'pr-35' into 0.0.3-draft 2026-03-27 02:26:52 +00:00
shoko
ca13a2e194 Merge branch 'pr-34' (URL encoding) into 0.0.3-draft 2026-03-27 02:26:41 +00:00
shoko
2c636048e7 security(polymarket-browse): improve response size limit with dynamic calculation
- Replace fixed 10MB limit with dynamic calculation
- get_max_response_size() computes limit based on PAGE_SIZE * multiplier
- Uses 10x multiplier (e.g., PAGE_SIZE=50 -> ~500KB * 10 = ~5MB)
- Clamped between 10MB minimum and 100MB maximum
- Formula: max(PAGE_SIZE * multiplier, 10MB) capped at 100MB
2026-03-27 01:36:32 +00:00
shoko
3016d1287c test(polymarket-browse): add URL encoding unit tests
Add TestUrlEncoding class testing quote() encodes:
- Space -> %20
- & -> %26
- = -> %3D
- % -> %25
- + -> %2B
- ( -> %28
- ) -> %29
- # -> %23
2026-03-27 01:14:45 +00:00
shoko
350fe17e87 docs(polymarket-browse): create SECURITY.md tracking audit findings
- Document fixed security issues from 2026-03-25 audit
- Track all 7 security issues and their fixes
- Add reporting instructions
2026-03-26 19:17:16 +00:00
shoko
32ed72868b docs(polymarket-browse): update SKILL.md with troubleshooting, examples, changelog, credits
- Clarify --detail argument (only for MATCH markets, error on out of range)
- Add Troubleshooting section with common issues
- Add Examples section with usage patterns
- Add Changelog section
- Add Credits section
2026-03-26 19:16:19 +00:00
shoko
c0484ab340 security(polymarket-browse): add token bucket rate limiter for API calls
- Add RateLimiter class (token bucket algorithm)
- Thread-safe for use with ThreadPoolExecutor
- RATE_LIMIT_CALLS = 10 per RATE_LIMIT_WINDOW = 1 second
- _rate_limiter.acquire() called before each API request
- Also add MAX_RESPONSE_SIZE check in fetch_page
2026-03-26 19:15:02 +00:00
shoko
36a7e8b3eb security(polymarket-browse): add MAX_RESPONSE_SIZE limit to prevent memory exhaustion
- Add MAX_RESPONSE_SIZE = 10MB constant
- Check response size before json.loads() in fetch_page()
- Raises ValueError if response exceeds limit
- Prevents memory exhaustion from malicious/gigantic API responses
2026-03-26 19:13:13 +00:00
shoko
bb7eebf502 security(polymarket-browse): use proper URL encoding for --search parameter
- Import quote from urllib.parse
- Replace q.replace(' ', '%20') with quote(q, safe='')
- Properly encodes: &, =, %, +, #, ?, and other special chars
- Prevents URL injection attacks
2026-03-26 19:11:59 +00:00
3 changed files with 176 additions and 6 deletions

View File

@@ -1,6 +1,6 @@
--- ---
name: polymarket-browse name: polymarket-browse
version: 0.0.2 version: 0.0.3
category: research category: research
description: Browse tradeable Polymarket events by game category. Shows active matches with ML odds (cents format), volume, tournament, and market URLs. Supports Counter Strike, League of Legends, Dota 2, Valorant, NBA, NFL, UFC, Tennis. description: Browse tradeable Polymarket events by game category. Shows active matches with ML odds (cents format), volume, tournament, and market URLs. Supports Counter Strike, League of Legends, Dota 2, Valorant, NBA, NFL, UFC, Tennis.
--- ---
@@ -47,7 +47,11 @@ polymarket-browse [--category "Counter Strike"] [--limit 5] [--matches N] [--non
- `--search` : Free-text team/term search within the selected category. Appends to the category query. Example: `--category "Counter Strike" --search "FlyQuest"` - `--search` : Free-text team/term search within the selected category. Appends to the category query. Example: `--category "Counter Strike" --search "FlyQuest"`
- `--matches-only` : Show only match markets (suppress non-match section). - `--matches-only` : Show only match markets (suppress non-match section).
- `--non-matches-only` : Show only non-match markets (suppress match section). - `--non-matches-only` : Show only non-match markets (suppress match section).
- `--detail` : Index of match event (1-indexed) to show detailed markets. Default: 1. Set to 0 to disable. - `--detail N` : Show detailed markets for match event N (1-indexed).
- Only applies to MATCH markets (not non-match/tournament markets)
- Default: 1 (auto-shows details for first match)
- Set to 0 to disable detail view
- If N exceeds available matches, shows error with available range
- `--list-categories` : List available game categories and exit - `--list-categories` : List available game categories and exit
- `--raw` : Show all events without tradeable filter (for debugging). Includes fetch stats. - `--raw` : Show all events without tradeable filter (for debugging). Includes fetch stats.
- `--no-cache` : Disable caching and fetch fresh data from the API. - `--no-cache` : Disable caching and fetch fresh data from the API.
@@ -138,9 +142,12 @@ The script first fetches page 1 to determine total pages, then fetches remaining
## Rate Limiting ## Rate Limiting
- Exponential backoff: 2s → 4s → 8s → 16s → 32s - TokenBucket rate limiter: 10 API calls per second
- Exponential backoff on retries: 2s → 4s → 8s → 16s → 32s
- Max 5 retries before aborting - Max 5 retries before aborting
**URL Encoding**: Special characters in `--search` (e.g., `&`, `=`, `%`, `+`, `#`) are properly encoded to prevent URL injection.
## Caching ## Caching
Results are cached in `~/.cache/polymarket-browse/` with a **5-minute TTL** to reduce redundant API calls. Results are cached in `~/.cache/polymarket-browse/` with a **5-minute TTL** to reduce redundant API calls.
@@ -155,3 +162,91 @@ All odds are shown in **cents** format:
- `30c` = 0.30 probability - `30c` = 0.30 probability
- `95c` = 0.95 probability - `95c` = 0.95 probability
- `GamerLegion 28c | 72c Team Yandex` = GamerLegion at 28c, Team Yandex at 72c - `GamerLegion 28c | 72c Team Yandex` = GamerLegion at 28c, Team Yandex at 72c
## Troubleshooting
### "WARNING: Partial fetch" appears
The API returned incomplete data due to an error/timeout. Results shown may be incomplete. Try again with `--no-cache` to force a fresh fetch.
### No markets appear
- Verify your category is correct: `--list-categories`
- Try with `--raw` to see all events (not just tradeable ones)
- Some categories may have no active match markets at certain times
### Why did my match disappear?
Matches are filtered out when:
- They have ended (startTime > 4 hours ago)
- BO2 matches ended in a tie (1-1)
- The market has converged (bestBid >= 0.99 or bestAsk <= 0.01)
- The event has ended (endDate passed)
### Telegram not working
- Verify `BOT_TOKEN` and `CHAT_ID` environment variables are set
- Ensure bot is started and chat ID is correct
- Check Telegram has not blocked the bot
## Examples
### Basic usage
```bash
# Browse Counter Strike matches (default)
polymarket-browse
# Browse NBA matches
polymarket-browse --category NBA
# Show more results
polymarket-browse --limit 10
```
### Searching for teams
```bash
# Find FlyQuest Counter Strike matches
polymarket-browse --category "Counter Strike" --search "FlyQuest"
# Find any team/event across category
polymarket-browse --category "Counter Strike" --search "Spirit"
```
### Filtering results
```bash
# Show only match markets (no tournament futures)
polymarket-browse --matches-only
# Show only non-match markets (tournaments, props)
polymarket-browse --non-matches-only
# Different limits for each section
polymarket-browse --matches 10 --non-matches 5
```
### Using --detail
```bash
# Show details for 1st match (default behavior, auto-enabled)
polymarket-browse --detail 1
# Show details for 3rd match
polymarket-browse --detail 3
# Disable detail view
polymarket-browse --detail 0
```
### Debugging
```bash
# Show all events without tradeable filter
polymarket-browse --raw
# Force fresh data (bypass cache)
polymarket-browse --no-cache
# Limit total events for quick snapshot
polymarket-browse --max-total 20
```
### Timezone
```bash
# Display times in different timezone (default: UTC+7/WIB)
polymarket-browse --timezone UTC+8
polymarket-browse --timezone UTC-5
```

View File

@@ -11,10 +11,11 @@ import time
import argparse import argparse
import hashlib import hashlib
import os import os
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone, timedelta from datetime import datetime, timezone, timedelta
from typing import Any, Callable, TypedDict from typing import Any, Callable, TypedDict
from urllib.parse import urlencode from urllib.parse import urlencode, quote
from urllib.request import urlopen, Request from urllib.request import urlopen, Request
@@ -98,6 +99,11 @@ class FetchResult(TypedDict):
PAGE_SIZE = 50 PAGE_SIZE = 50
MAX_RETRIES = 5 MAX_RETRIES = 5
INITIAL_RETRY_DELAY = 2 # exponential backoff starts at 2s INITIAL_RETRY_DELAY = 2 # exponential backoff starts at 2s
MAX_RESPONSE_SIZE_MULTIPLIER = 10 # Response size limit = PAGE_SIZE * multiplier
MAX_RESPONSE_SIZE_MIN = 10 * 1024 * 1024 # 10MB minimum
MAX_RESPONSE_SIZE_MAX = 100 * 1024 * 1024 # 100MB maximum for safety
RATE_LIMIT_CALLS = 10 # max API calls
RATE_LIMIT_WINDOW = 1.0 # per second
WIB = timezone(timedelta(hours=7)) # UTC+7 for Indonesian users WIB = timezone(timedelta(hours=7)) # UTC+7 for Indonesian users
_DISPLAY_TZ = WIB # Module-level timezone for display (configurable via --timezone) _DISPLAY_TZ = WIB # Module-level timezone for display (configurable via --timezone)
@@ -139,6 +145,48 @@ def parse_timezone(tz_str: str) -> timezone:
return WIB return WIB
def get_max_response_size(page_size: int = PAGE_SIZE) -> int:
"""
Calculate max response size based on expected payload.
Uses 10x multiplier: if PAGE_SIZE=50 events, expected ~500KB-5MB,
so 10x gives 5MB-50MB. Clamped between 10MB and 100MB.
"""
multiplier = MAX_RESPONSE_SIZE_MULTIPLIER * page_size * 1024 # rough estimate
size = max(multiplier, MAX_RESPONSE_SIZE_MIN)
return min(size, MAX_RESPONSE_SIZE_MAX)
class RateLimiter:
"""Token bucket rate limiter for API calls. Thread-safe for use with ThreadPoolExecutor."""
def __init__(
self, calls: int = RATE_LIMIT_CALLS, window: float = RATE_LIMIT_WINDOW
):
self.calls = calls
self.window = window
self.tokens = float(calls)
self.last_update = time.monotonic()
self._lock = threading.Lock()
def acquire(self) -> None:
"""Block until a token is available."""
with self._lock:
now = time.monotonic()
elapsed = now - self.last_update
self.tokens = min(
self.calls, self.tokens + elapsed * (self.calls / self.window)
)
if self.tokens < 1:
wait_time = (1 - self.tokens) * (self.window / self.calls)
time.sleep(wait_time)
self.tokens = 0
else:
self.tokens -= 1
self.last_update = time.monotonic()
_rate_limiter = RateLimiter()
GAME_CATEGORIES = { GAME_CATEGORIES = {
"All Esports": "Esports", "All Esports": "Esports",
"Counter Strike": "Counter Strike", "Counter Strike": "Counter Strike",
@@ -206,7 +254,7 @@ def fetch_page(
) -> dict[str, Any] | None: ) -> dict[str, Any] | None:
base = "https://gamma-api.polymarket.com/public-search" base = "https://gamma-api.polymarket.com/public-search"
url = ( url = (
f"{base}?q={q.replace(' ', '%20')}&limit={PAGE_SIZE}&page={page}" f"{base}?q={quote(q, safe='')}&limit={PAGE_SIZE}&page={page}"
f"&search_profiles=false&search_tags=false" f"&search_profiles=false&search_tags=false"
f"&keep_closed_markets=0&events_status=active&cache=false" f"&keep_closed_markets=0&events_status=active&cache=false"
) )
@@ -216,9 +264,16 @@ def fetch_page(
if attempt > 0: if attempt > 0:
time.sleep(delay) time.sleep(delay)
try: try:
_rate_limiter.acquire()
req = Request(url, headers={"User-Agent": "Mozilla/5.0"}) req = Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urlopen(req, timeout=10) as r: with urlopen(req, timeout=10) as r:
return json.loads(r.read()) data = r.read()
max_size = get_max_response_size(PAGE_SIZE)
if len(data) > max_size:
raise ValueError(
f"API response too large: {len(data)} bytes (max {max_size})"
)
return json.loads(data)
except Exception: except Exception:
if attempt < max_retries - 1: if attempt < max_retries - 1:
delay *= 2 delay *= 2

View File

@@ -1969,5 +1969,25 @@ class TestTimezoneParsing(unittest.TestCase):
self.assertEqual(tz, timezone(timedelta(hours=7))) self.assertEqual(tz, timezone(timedelta(hours=7)))
class TestUrlEncoding(unittest.TestCase):
"""Tests for proper URL encoding of search queries."""
def test_quote_encodes_special_chars(self):
"""quote() should properly encode all special characters."""
from urllib.parse import quote
test_cases = [
("Team A", "Team%20A"),
("Team A & Team B", "Team%20A%20%26%20Team%20B"),
("a=b", "a%3Db"),
("100%", "100%25"),
("C++", "C%2B%2B"),
("Team (A)", "Team%20%28A%29"),
("Team#1", "Team%231"),
]
for input_str, expected in test_cases:
self.assertEqual(quote(input_str, safe=""), expected)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()