Track Indian portfolio holdings, create snapshots, and generate weekly news digests.
When Joseph provides holding updates (quantity changes, new purchases, avg price changes) and wants to maintain the tracking file and generate a dated portfolio snapshot. Also when generating the weekly portfolio news digest (every Sunday via cron). This is the recurring maintenance workflow — distinct from quarterly analysis (see portfolio-analysis).
wiki/wealth/invest/portfolio/holdings.csv — master holdings (source, code, ticker, name, qty, cost_price, asset_class)wiki/wealth/invest/portfolio/tracking/YYYY-MM-DD-portfolio.md — dated snapshotswiki/wealth/invest/portfolio/news/YYYY-Wnn-weekly-news.md — weekly news digests (ISO week number)The portfolio snapshot runs weekly via cron (Sundays 13:30 UTC).
de96312284a9 ("Portfolio Weekly Snapshot")python3 /opt/data/scripts/portfolio-report.pyslack:C0BNN2Z950V30 13 0 (weekly, Sunday)holdings.csv, fetches Yahoo Finance (stocks/ETFs) + AMFI (MFs), writes YYYY-MM-DD-portfolio.md, outputs Slack-formatted summary to stdout.To test without writing wiki files: python3 /opt/data/scripts/portfolio-report.py --dry-run
To debug delivery failures: check cronjob_manage(action='list') → look at last_status, last_error, last_delivery_error. Then read the job JSON at /opt/data/cron/jobs.json for full error context.
When hledger has been updated with new transactions:
1. Run sync script: python3 /opt/data/scripts/update-holdings.py
2. Script queries hledger for quantities (commodity balances) and computes per-person cost prices dynamically from transaction data
3. Updates tracking/holdings.csv with new schema (source, holder, code, name, qty, cost_price, asset_class, yahoo_symbol, exchange, amfi_code)
4. Verify: spot-check 3 holdings, confirm Stock costs: N person:stock combinations is non-zero in script output
Frequency: once or twice a month (Joseph invests, doesn't trade). Run before weekly portfolio cron.
How it works:hledger (source of truth) → update-holdings.py → tracking/holdings.csv → portfolio-report.py → portfolio values
1. Queries hledger for quantities per person per fund/stock (commodity balances)
2. Computes per-person cost prices from hledger -B flag (MFs) and journal lot parsing (stocks)
3. Enriches with metadata (name, yahoo_symbol, amfi_code, exchange) from hardcoded lookup tables
4. Adds FD/Bond flat entries (hardcoded until hledger import)
5. Writes updated tracking/holdings.csv
When Joseph provides holding changes directly:
1. Parse Joseph's update: fund/stock name, new units, new avg price, total invested value
2. Locate the matching row in holdings.csv (match by name or scheme code)
3. Patch the row: update qty and cost_price columns
4. Verify the patch landed correctly (read back the affected lines)
holdings.csv format:source,code,nse_ticker,yahoo_symbol,name,qty,cost_price,asset_class
1. Verify holdings.csv has correct FD/Bond entries with current values: grep 'fd\|bond' tracking/holdings.csv
2. Verify ITCHOTELS (or any new demerged stock) appears: grep ITCHOTELS tracking/holdings.csv
3. Spot-check 3 holdings: qty and cost_price match hledger
4. Then run python3 /opt/data/scripts/portfolio-report.py to regenerate the portfolio file
portal.amfiindia.com/spages/NAVAll.txt (semicolon-delimited)Batch-fetch all tickers in one script. Do not make serial curl calls (times out).
1. Read holdings.csv for all quantities and cost prices
2. Compute weighted-average buy price for stocks held in both ICICI and Zerodha (e.g., Coal India, GAIL, NMDC)
3. Value = qty × CMP (stocks/ETFs) or units × NAV (MFs)
4. P&L = value - (qty × cost_price)
5. Asset allocation: stocks %, ETFs % (with gold/silver/index sub-breakdown), MFs %
6. Write YYYY-MM-DD-portfolio.md following the template (see references/snapshot-template.md)
deliver value in cron jobs MUST be platform:chat_id (e.g., slack:C0BNN2Z950V), not a bare channel ID.blocked_config error: "delivery platform 'C0BNN2Z950V' is not a known cron delivery target".:thread_id segment lands in the main chat, not a topic.1. GET https://fc.yahoo.com (ignore error, just need cookie)
2. GET https://query2.finance.yahoo.com/v1/test/getcrumb (returns crumb string)
3. Use http.cookiejar + urllib.request.build_opener to maintain session
4. Fetch: https://query2.finance.yahoo.com/v8/finance/chart/{ticker}?range=1d&interval=1d&crumb={crumb}
period1/period2 timestamps instead of range=1dhttps://portal.amfiindia.com/spages/NAVAll.txtScheme Code;ISIN Payout;ISIN Reinvest;Scheme Name;Plan;Option;NAV;Dateawk for speed (not Python line-by-line)cost_price is average buy price, not current pricestock, etf, mf, fd, bondupdate-holdings.py to syncstat) to see if it was modified between the two snapshot dates, (3) check what update-holdings.py would produce — ICICI stocks use hardcoded ICICI_COST_PRICES (line ~243), Zerodha stocks use journal-parsed costs via get_all_stock_costs(). If the CSV value differs from what the script would produce, the CSV was manually edited. (4) Verify by running python3 /opt/data/scripts/update-holdings.py — it will overwrite with the script's values.current_value column in tracking/holdings.csv for accrued/maturity values. portfolio-report.py reads this column for FDs/bonds instead of falling back to cost. Current values must be updated manually in the hardcoded entries in update-holdings.py (FD accrued values change over time). The weekly cron uses portfolio-report.py which pulls live prices for stocks/MFs and uses current_value for FDs/bonds — no OCR screenshots needed.portfolio/holdings.csv (old schema, no holder column, used by sync-holdings.py) and portfolio/tracking/holdings.csv (new schema with holder column, used by portfolio-report.py and the cron). The weekly cron reads from tracking/holdings.csv.When the portfolio file's "Price" (invested) column doesn't match the broker dashboard's invested amount:
1. Stocks match closely — differences under ₹5K per stock are normal (brokerage/taxes in broker's calculation vs hledger's clean cost).
2. MFs diverge significantly — the cost_price in holdings.csv is a weighted average from hledger transactions. Zerodha calculates "invested" from actual transaction-level costs (including charges). For funds with many SIPs, the weighted average can diverge by ₹4–5L per person.
3. Total invested may still match — the portfolio file total and Zerodha total can agree even when per-person breakdowns differ. This means the issue is cost allocation across accounts, not missing transactions.
4. Diagnosis steps:
a. Compute qty × cost_price per holder from tracking/holdings.csv (use Python, not mental math — fractional units matter)
b. Compare against broker dashboard per-account invested amounts
c. Stocks will match closely; large gaps point to MF cost_price divergence
d. Re-sync with python3 /opt/data/scripts/update-holdings.py to pull fresh hledger quantities and per-person cost prices
e. If gap persists after re-sync, the hledger MF transactions may be missing or have incorrect cost entries — check the investment journals directly
5. The Individual Holdings table may show rounded quantities while calculations use exact fractional units from holdings.csv — this is cosmetic, not a data error.
6. Duplicate flat INR entries from old imports — when convert-zerodha-stocks.py generates commodity entries (with per-unit cost) from broker CSVs, the OLD flat INR entries for the same transactions may still exist in investments.journal for ANY year (2024, 2025, etc.). Both are included via main.journal, so hledger double-counts: the INR balance is inflated by the flat amounts, while commodity quantities are correct. Symptom: portfolio file's invested amount per person is higher than Zerodha dashboard. Fix: remove ALL flat INR entries (both BUYs and Sells) from the relevant investments.journal files — check every year's file, not just the most recent. After removal, update ALL subsequent balance assertions in balances.journal for the affected bank account: each assertion must be adjusted by the net amount of removed entries (sum of BUY debits minus SELL credits). Verify: hledger -f journals/main.journal bal Assets:Invest:Stocks:Zerodha:<Person> --no-total should show only commodity balances, no flat INR amounts. Also run hledger -f journals/main.journal bal to confirm no balance assertion errors.
7. MF redemptions recorded as flat INR — when MF sell/redemption transactions are imported as flat INR amounts to the parent account (Assets:Invest:MFs:Zerodha:Rani) instead of as commodity-level sells to the specific fund (Rani:AX-LF-DG), the fund's unit balance stays at the pre-sale quantity. The portfolio file will count redeemed units as still held. Symptom: large per-person invested gap between portfolio file and Zerodha dashboard, especially for funds with partial redemptions. Fix: convert flat INR redemption entries to proper commodity sells with correct unit quantities and lot costs from the source CSV (the CSV has quantity and price fields for sells).
8. Hardcoded cost prices shared across holders are wrong — the update-holdings.py script now computes per-person cost prices dynamically from hledger transaction data (-B flag for MFs, journal parsing for stocks). If the script still uses hardcoded COST_PRICES for Zerodha holdings, costs will be wrong because different people bought at different times/prices. Verify: the script output should show Stock costs: N person:stock combinations (non-zero). If it shows 0, the journal parsing regex needs fixing.
9. hledger -B shows phantom buy cost for fully-sold positions — when a commodity position is completely sold (qty=0), hledger bal -B on the sub-account may still display the original buy cost as INR while the commodity balance is 0 or negative. Do NOT sum sub-account -B balances to get total cost for an account with fully-sold positions — use only the active sub-accounts, or use the parent account's INR balance. Symptom: portfolio invested total is higher than broker dashboard because phantom costs inflate the sum.
10. Zerodha "invested" includes transaction charges; hledger does not — hledger records the clean share price per unit (the commodity line {PRICE INR}). Zerodha's dashboard "invested" includes STT (0.1% buy-side for equity), stamp duty (0.015%), exchange transaction charges, IPFT, SEBI charges, and GST on brokerage. This creates a persistent gap: Zerodha_invested - hledger_cost ≈ clean_cost × 0.12%. The gap is proportional to total stock value — small for small portfolios (₹1–2K on ₹10L, invisible after OCR rounding), larger for bigger ones (₹7–8K on ₹25L). Both JJU and Rani are affected; it's just invisible for Rani because her stock portfolio is smaller.
11. Tax P&L reports (CSV and HTML) only have sell-side charges — the Zerodha Tax P&L reports (taxpnl-.csv and taxpnl-.html) contain charge breakdowns (brokerage, STT, stamp duty, exchange charges, GST) only for closed/sold positions. The HTML version additionally has "Other Credits & Debits" (AMC, DP charges, delayed payment) but these are account-level, not transaction-level. Neither format has buy-side charges for open holdings. To reconcile the gap from pitfall #10, you need Contract Notes from Zerodha Console → Reports → Contract Notes. Contract Notes have per-trade charge breakdowns on both buys and sells.
12. Portfolio file "invested" comes from Zerodha OCR, not hledger — compile-portfolio.py reads Zerodha dashboard screenshots via OCR and uses the "Price" (invested) column directly. It does NOT compute invested from hledger. This means the portfolio file's "invested" will always include Zerodha's transaction charges, while hledger-based calculations (holdings.csv cost_price × qty) will not. When comparing, expect a gap equal to total transaction charges across all holdings.
13. Corporate actions (demergers, splits, bonuses) may be missing from hledger — when a demergers allocates new shares (e.g., ITC Hotels from ITC demerger), Zerodha shows the new holding with a cost basis but hledger may not have the transaction. Symptom: total "invested" gap equals exactly the missing holding's invested amount. Diagnosis: screenshot the Zerodha dashboard by asset class (stocks/ETFs separately), compare each holding's invested against hledger's qty × cost_price. A holding present in Zerodha but absent from hledger is a missing import. Fix: add the corporate action as a commodity entry in zerodha-stocks.journal with a balancing Income:CorporateActions:Demerger line (no bank debit — shares are allocated free). Add the account to accounts.journal and the ticker to update-holdings.py STOCK_NAMES.
14. Best way to verify "invested" match: per-asset-class screenshots — don't rely on the total alone. Screenshot Zerodha dashboard for MFs and stocks/ETFs separately. Compare each holding's invested against qty × cost_price from holdings.csv. MFs typically match within rounding (Zerodha charges nil for direct plans). Stock gaps represent transaction charges. A gap that exactly equals a specific holding's value means that holding is missing from hledger.
15. Contract Notes (XML) have buy-side equity charges; Tax P&L does not — Zerodha Contract Notes (Console → Reports → Contract Notes) contain per-trade charge breakdowns on both buys and sells for equity trades. Tax P&L reports only have sell-side charges for closed positions. Contract Notes do NOT cover MFs (not exchange-traded). To reconcile the gap between hledger clean cost and Zerodha "invested", Contract Notes are the authoritative source for stock charges.
A weekly digest covering news for ALL holdings (stocks, ETFs, MFs). Complements the price snapshot — snapshot tracks values, digest tracks events.
Every Sunday morning via cron. Covers the Mon–Sat trading week.
1. Read holdings.csv for the full portfolio (stocks, ETFs, MFs).
2. Fetch price data via yfinance for all stock/ETF tickers (5-day period for WoW change).
3. Fetch news — yfinance .news attribute returns empty for Indian (.NS) tickers. Use web_search as fallback. Focus searches on significant movers (>2% WoW change) to avoid noise. Search queries: "{stock name} India stock news September 2026".
4. Get market context — Nifty 50, crude oil, gold, silver, Nasdaq 100 levels and WoW change via yfinance.
5. Write digest to wiki/wealth/invest/portfolio/news/YYYY-Wnn-weekly-news.md (date = ISO week number of today).
.news is empty for Indian tickers — always use web_search fallback. Don't assume the API will work.wiki/wealth/invest/portfolio/news/ — one file per weekYYYY-Wnn-weekly-news.md (ISO week number of run date)See references/news-digest-template.md for the markdown structure.
references/yahoo-finance-api.md — crumb auth, endpoints, rate limitsreferences/amfi-nav.md — NAV file format, scheme code lookupreferences/snapshot-template.md — portfolio snapshot markdown templatereferences/news-digest-template.md — weekly news digest markdown template