Feature/distill token optimization #3181
Open
+78
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Tomo1912
force-pushed
the
feature/distill-token-optimization
branch
5 times, most recently
from
4322b3e to
a3cff79
Compare
Add distill parameter to aggressively clean HTML before processing: - Remove scripts, styles, navigation, headers, footers - Remove ads, sidebars, popups, cookie banners - Remove social widgets and non-content elements - Normalize whitespace Typical token reduction: 60-85% This is an opt-in feature (distill=false by default) to maintain backward compatibility.
Tomo1912
force-pushed
the
feature/distill-token-optimization
branch
from
a3cff79 to
98c0dd3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add optional
distillparameter to the fetch server that aggressively cleans HTML content to minimize token usage. When enabled, removes scripts, styles, navigation, headers, footers, ads, and other non-essential elements before conversion to markdown. Achieves 72.8% average token reduction across real-world tests.Server Details
distillparameter tofetchtool)Motivation and Context
LLM token costs are a significant operational expense. Current
web-fetchreturns full HTML including navigation menus, ads, scripts, and UI clutter - wasting tokens on non-content elements.Test Results:
How Has This Been Tested?
distillparameter work unchangedBreaking Changes
None. The
distillparameter defaults toFalse, maintaining full backward compatibility.Types of changes
Checklist
Additional context
No new dependencies required - uses existing
readabilipyfor content extraction.