By the end of this tutorial, you’ll be able to:<\/p>\n\n\n\n
Time investment:<\/strong> 30 minutes setup, 2 minutes runtime for 100+ posts<\/p>\n\n\n\n Technical level:<\/strong> Basic command line knowledge required<\/p>\n\n\n\n Platform agnostic:<\/strong> This works for any blog platform (WordPress, Webflow, Ghost, custom CMS) as long as you have a sitemap.<\/p>\n\n\n\n Before writing any code, inspect your blog to understand:<\/p>\n\n\n\n Most blogs have a sitemap at:<\/p>\n\n\n\n Check your robots.txt file ( Visit one blog post and inspect the HTML (right-click \u2192 “View Page Source”):<\/p>\n\n\n\n Find these elements:<\/strong><\/p>\n\n\n\n Pro tip:<\/strong> Use browser DevTools (F12) to inspect elements and identify CSS selectors.<\/p>\n\n\n\n For example:<\/p>\n\n\n\n Create a Install dependencies:<\/p>\n\n\n\n What each library does:<\/strong><\/p>\n\n\n\n Create Critical:<\/strong> You must adjust the CSS selectors in Repeat for:<\/p>\n\n\n\n WordPress:<\/strong><\/p>\n\n\n\n Ghost:<\/strong><\/p>\n\n\n\n Medium:<\/strong><\/p>\n\n\n\n Expected output:<\/strong><\/p>\n\n\n\n Check the Good signs:<\/strong><\/p>\n\n\n\n Red flags:<\/strong><\/p>\n\n\n\n If you see red flags, revisit Step 4 and adjust your selectors.<\/p>\n\n\n\n Once you’re satisfied with the test:<\/p>\n\n\n\n For 100 posts, expect 1-2 minutes runtime.<\/p>\n\n\n\n Create This creates [Author’s note: This particular section is untested as of now.]<\/p>\n\n\n\n Want to keep your markdown files in sync as you publish new posts? Set up a cron job or GitHub Action.<\/p>\n\n\n\n Create Problem:<\/strong> Content div selector is wrong<\/p>\n\n\n\n Solution:<\/strong><\/p>\n\n\n\n Problem:<\/strong> Wrong Python command on Mac<\/p>\n\n\n\n Solution:<\/strong> Use Problem:<\/strong> Site blocking the scraper<\/p>\n\n\n\n Solution:<\/strong><\/p>\n\n\n\n Problem:<\/strong> Selectors don’t match your HTML structure<\/p>\n\n\n\n Solution:<\/strong> Update all selectors in Always use The script includes 0.5s delays between requests. Don’t remove these\u2014be a good internet citizen.<\/p>\n\n\n\n This lets you track content changes over time.<\/p>\n\n\n\n Keep a comment at the top of your script:<\/p>\n\n\n\n Run monthly to catch:<\/p>\n\n\n\n Converting your blog to markdown unlocks powerful AI-driven content analysis. What used to take days of manual auditing now takes minutes.<\/p>\n\n\n\n The key is getting clean, structured data that LLMs can parse effectively. With proper metadata extraction and YAML frontmatter, you can:<\/p>\n\n\n\n Time investment:<\/strong> 30 minutes setup.<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" What You’ll Learn By the end of this tutorial, you’ll be able to: Time investment: 30 minutes setup, 2 minutes runtime for 100+ posts Technical level: Basic command line knowledge required Prerequisites Platform agnostic: This works for any blog platform (WordPress, Webflow, Ghost, custom CMS) as long as you have a sitemap. Step 1: Understand…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","footnotes":""},"categories":[8],"tags":[],"class_list":["post-74","post","type-post","status-publish","format-standard","hentry","category-random"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/posts\/74","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/comments?post=74"}],"version-history":[{"count":8,"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/posts\/74\/revisions"}],"predecessor-version":[{"id":93,"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/posts\/74\/revisions\/93"}],"wp:attachment":[{"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/media?parent=74"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/categories?post=74"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sujandeswal.com\/blog\/wp-json\/wp\/v2\/tags?post=74"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Prerequisites<\/h2>\n\n\n\n
\n
\n\n\n\nStep 1: Understand Your Blog Structure<\/h2>\n\n\n\n
A. Sitemap Location<\/h3>\n\n\n\n
\n
https:\/\/yourblog.com\/sitemap.xml<\/code><\/li>\n\n\n\nhttps:\/\/yourblog.com\/sitemap_index.xml<\/code><\/li>\n\n\n\nhttps:\/\/yourblog.com\/post-sitemap.xml<\/code><\/li>\n<\/ul>\n\n\n\nyourblog.com\/robots.txt<\/code>) to find your sitemap URL.<\/p>\n\n\n\nB. HTML Structure<\/h3>\n\n\n\n
\n
post-content<\/code>, article-body<\/code>, rich-text<\/code>)<\/li>\n\n\n\n<h1><\/code> tag)<\/li>\n\n\n\n\n
<div class=\"text-rich-text\"><\/code><\/li>\n\n\n\n<h1 class=\"heading-style-h2\"><\/code><\/li>\n\n\n\n<p class=\"text-size-regular text-color-brandnile\"><\/code><\/li>\n\n\n\n<p class=\"text-size-regular text-color-grey\"><\/code><\/li>\n<\/ul>\n\n\n\nStep 2: Set Up Your Environment<\/h2>\n\n\n\n
Create Project Folder<\/h3>\n\n\n\n
mkdir blog_converter\ncd blog_converter<\/code><\/pre>\n\n\n\nInstall Python Dependencies<\/h3>\n\n\n\n
requirements.txt<\/code> file:<\/p>\n\n\n\nrequests>=2.31.0\nbeautifulsoup4>=4.12.0\nlxml>=4.9.0\nmarkdownify>=0.11.6\nPyYAML>=6.0.1<\/code><\/pre>\n\n\n\npip install -r requirements.txt\n# Or on Mac: pip3 install -r requirements.txt<\/code><\/pre>\n\n\n\n\n
requests<\/code> – Fetches web pages<\/li>\n\n\n\nbeautifulsoup4<\/code> – Parses HTML<\/li>\n\n\n\nlxml<\/code> – Fast XML\/HTML parser<\/li>\n\n\n\nmarkdownify<\/code> – Converts HTML to Markdown<\/li>\n\n\n\nPyYAML<\/code> – Handles YAML frontmatter<\/li>\n<\/ul>\n\n\n\nStep 3: Build the Converter Script<\/h2>\n\n\n\n
blog_to_markdown.py<\/code>:<\/p>\n\n\n\n#!\/usr\/bin\/env python3\n\"\"\"\nBlog Post to Markdown Converter\nExtracts blog posts from sitemap and converts them to markdown with YAML frontmatter\n\"\"\"\n\nimport re\nimport requests\nfrom bs4 import BeautifulSoup\nfrom markdownify import markdownify as md\nfrom pathlib import Path\nfrom datetime import datetime\nimport yaml\nimport time\n\nclass BlogConverter:\n def __init__(self, sitemap_url, output_dir=\"blog_posts\"):\n self.sitemap_url = sitemap_url\n self.output_dir = Path(output_dir)\n self.output_dir.mkdir(exist_ok=True)\n self.session = requests.Session()\n self.session.headers.update({\n 'User-Agent': 'Mozilla\/5.0 (compatible; BlogConverter\/1.0)'\n })\n\n def fetch_sitemap(self):\n \"\"\"Fetch and parse the sitemap to get all blog post URLs\"\"\"\n print(f\"Fetching sitemap from {self.sitemap_url}...\")\n response = self.session.get(self.sitemap_url)\n response.raise_for_status()\n\n soup = BeautifulSoup(response.content, 'xml')\n urls = soup.find_all('loc')\n\n # Filter only blog post URLs (adjust pattern for your blog)\n blog_urls = [url.text for url in urls if '\/blog-post\/' in url.text]\n print(f\"Found {len(blog_urls)} blog posts\")\n return blog_urls\n\n def fetch_page(self, url):\n \"\"\"Fetch a single blog post page\"\"\"\n response = self.session.get(url)\n response.raise_for_status()\n return BeautifulSoup(response.content, 'html.parser')\n\n def extract_metadata(self, soup, url):\n \"\"\"Extract metadata from the blog post\"\"\"\n metadata = {\n 'url': url,\n 'slug': url.split('\/blog-post\/')[-1]\n }\n\n # Title - adjust selector for your blog\n title_tag = soup.find('h1', class_='heading-style-h2')\n metadata['title'] = title_tag.get_text(strip=True) if title_tag else 'Untitled'\n\n # Meta description\n meta_desc = soup.find('meta', attrs={'name': 'description'})\n metadata['description'] = meta_desc['content'] if meta_desc else ''\n\n # Category - adjust selector for your blog\n category_div = soup.find('div', class_='w-dyn-list')\n if category_div:\n category_item = category_div.find('div', role='listitem')\n if category_item:\n metadata['category'] = category_item.get_text(strip=True)\n\n # Author - adjust selector for your blog\n author_tag = soup.find('p', class_='text-size-regular text-color-brandnile')\n metadata['author'] = author_tag.get_text(strip=True) if author_tag else ''\n\n # Dates - adjust selector for your blog\n date_divs = soup.find_all('p', class_='text-size-regular text-color-grey')\n for div in date_divs:\n text = div.get_text(strip=True)\n parent = div.find_parent('div', class_='date_flex')\n if parent:\n label = parent.find('div', class_='text-weight-medium')\n if label:\n label_text = label.get_text(strip=True)\n if 'Published' in label_text:\n metadata['published_date'] = text\n elif 'Updated' in label_text:\n metadata['updated_date'] = text\n\n # Featured image\n img_tag = soup.find('img', class_='blogtp_hero-banner')\n if img_tag:\n metadata['featured_image'] = img_tag.get('src', '')\n\n # OG Image (fallback)\n if not metadata.get('featured_image'):\n og_image = soup.find('meta', property='og:image')\n if og_image:\n metadata['featured_image'] = og_image['content']\n\n return metadata\n\n def extract_content(self, soup):\n \"\"\"Extract the main blog post content\"\"\"\n # Adjust selector for your blog's content container\n content_div = soup.find('div', class_='text-rich-text')\n\n if not content_div:\n return \"\"\n\n # Remove unwanted elements\n for element in content_div.find_all(['script', 'style']):\n element.decompose()\n\n # Convert to markdown\n html_content = str(content_div)\n markdown_content = md(html_content, heading_style=\"ATX\", bullets=\"-\")\n\n # Clean up the markdown\n markdown_content = self.clean_markdown(markdown_content)\n\n return markdown_content\n\n def clean_markdown(self, text):\n \"\"\"Clean up markdown formatting\"\"\"\n # Remove excessive newlines\n text = re.sub(r'\\n{3,}', '\\n\\n', text)\n\n # Fix spacing around headers\n text = re.sub(r'\\n(#{1,6} )', r'\\n\\n\\1', text)\n text = re.sub(r'(#{1,6} .+)\\n', r'\\1\\n\\n', text)\n\n # Remove leading\/trailing whitespace\n text = text.strip()\n\n return text\n\n def create_markdown_file(self, metadata, content):\n \"\"\"Create a markdown file with YAML frontmatter\"\"\"\n # Create frontmatter\n frontmatter = {\n 'title': metadata.get('title', ''),\n 'slug': metadata.get('slug', ''),\n 'url': metadata.get('url', ''),\n 'description': metadata.get('description', ''),\n 'author': metadata.get('author', ''),\n 'category': metadata.get('category', ''),\n 'published_date': metadata.get('published_date', ''),\n 'updated_date': metadata.get('updated_date', ''),\n 'featured_image': metadata.get('featured_image', '')\n }\n\n # Remove empty values\n frontmatter = {k: v for k, v in frontmatter.items() if v}\n\n # Create the complete markdown document\n markdown_doc = \"---\\n\"\n markdown_doc += yaml.dump(frontmatter, allow_unicode=True, sort_keys=False)\n markdown_doc += \"---\\n\\n\"\n markdown_doc += content\n\n return markdown_doc\n\n def save_markdown(self, slug, markdown_content):\n \"\"\"Save the markdown file\"\"\"\n # Sanitize filename\n filename = re.sub(r'[^\\w\\-]', '_', slug)\n filepath = self.output_dir \/ f\"{filename}.md\"\n\n with open(filepath, 'w', encoding='utf-8') as f:\n f.write(markdown_content)\n\n return filepath\n\n def process_post(self, url):\n \"\"\"Process a single blog post\"\"\"\n try:\n print(f\"Processing: {url}\")\n\n # Fetch the page\n soup = self.fetch_page(url)\n\n # Extract metadata\n metadata = self.extract_metadata(soup, url)\n\n # Extract content\n content = self.extract_content(soup)\n\n if not content:\n print(f\" \u26a0\ufe0f Warning: No content found for {url}\")\n return None\n\n # Create markdown document\n markdown_doc = self.create_markdown_file(metadata, content)\n\n # Save to file\n filepath = self.save_markdown(metadata['slug'], markdown_doc)\n\n print(f\" \u2713 Saved to: {filepath}\")\n return filepath\n\n except Exception as e:\n print(f\" \u2717 Error processing {url}: {str(e)}\")\n return None\n\n def create_index(self, results):\n \"\"\"Create an index file with all posts\"\"\"\n index_path = self.output_dir \/ \"INDEX.md\"\n\n with open(index_path, 'w', encoding='utf-8') as f:\n f.write(\"# Blog Posts Index\\n\\n\")\n f.write(f\"Total posts: {len(results)}\\n\\n\")\n f.write(f\"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\\n\\n\")\n f.write(\"---\\n\\n\")\n\n for i, (url, filepath) in enumerate(results.items(), 1):\n if filepath:\n slug = filepath.stem\n f.write(f\"{i}. [{slug}]({filepath.name}) - {url}\\n\")\n\n print(f\"\\n\u2713 Created index file: {index_path}\")\n\n def run(self, limit=None):\n \"\"\"Run the conversion process\"\"\"\n start_time = time.time()\n\n # Fetch all blog URLs\n blog_urls = self.fetch_sitemap()\n\n # Limit for testing\n if limit:\n blog_urls = blog_urls[:limit]\n print(f\"Processing first {limit} posts only (testing mode)\")\n\n # Process each post\n results = {}\n for i, url in enumerate(blog_urls, 1):\n print(f\"\\n[{i}\/{len(blog_urls)}]\")\n filepath = self.process_post(url)\n results[url] = filepath\n\n # Be polite - add a small delay between requests\n time.sleep(0.5)\n\n # Create index\n self.create_index(results)\n\n # Summary\n successful = sum(1 for fp in results.values() if fp is not None)\n failed = len(results) - successful\n elapsed = time.time() - start_time\n\n print(f\"\\n{'='*60}\")\n print(f\"Conversion Complete!\")\n print(f\"{'='*60}\")\n print(f\"Total posts: {len(results)}\")\n print(f\"Successful: {successful}\")\n print(f\"Failed: {failed}\")\n print(f\"Time elapsed: {elapsed:.2f} seconds\")\n print(f\"Output directory: {self.output_dir.absolute()}\")\n\n\ndef main():\n import argparse\n\n parser = argparse.ArgumentParser(description='Convert blog posts to Markdown')\n parser.add_argument('--sitemap', required=True,\n help='Sitemap URL')\n parser.add_argument('--output', default='blog_posts',\n help='Output directory')\n parser.add_argument('--limit', type=int,\n help='Limit number of posts (for testing)')\n\n args = parser.parse_args()\n\n converter = BlogConverter(args.sitemap, args.output)\n converter.run(limit=args.limit)\n\n\nif __name__ == '__main__':\n main()<\/code><\/pre>\n\n\n\nStep 4: Customize for Your Blog<\/h2>\n\n\n\n
extract_metadata()<\/code> and extract_content()<\/code> methods to match your blog’s HTML structure.<\/p>\n\n\n\nHow to Find Your Selectors:<\/h3>\n\n\n\n
\n
post-title<\/code>, entry-title<\/code>, article-heading<\/code>)<\/li>\n\n\n\n# Change this:\ntitle_tag = soup.find('h1', class_='heading-style-h2')\n\n# To your selector:\ntitle_tag = soup.find('h1', class_='post-title')<\/code><\/pre>\n\n\n\n\n
Common Selectors by Platform:<\/h3>\n\n\n\n
content_div = soup.find('div', class_='entry-content')\ntitle_tag = soup.find('h1', class_='entry-title')\nauthor_tag = soup.find('span', class_='author')<\/code><\/pre>\n\n\n\ncontent_div = soup.find('div', class_='post-content')\ntitle_tag = soup.find('h1', class_='post-title')\nauthor_tag = soup.find('a', class_='author-name')<\/code><\/pre>\n\n\n\ncontent_div = soup.find('article')\ntitle_tag = soup.find('h1')\nauthor_tag = soup.find('a', attrs={'data-action': 'show-user-card'})<\/code><\/pre>\n\n\n\nStep 5: Run the Converter<\/h2>\n\n\n\n
Test with Limited Posts First<\/h3>\n\n\n\n
python3 blog_to_markdown.py --sitemap https:\/\/yourblog.com\/sitemap.xml --limit 3<\/code><\/pre>\n\n\n\nFetching sitemap from https:\/\/yourblog.com\/sitemap.xml...\nFound 150 blog posts\nProcessing first 3 posts only (testing mode)\n\n[1\/3]\nProcessing: https:\/\/yourblog.com\/blog-post\/example-post\n \u2713 Saved to: blog_posts\/example-post.md\n\n[2\/3]\nProcessing: https:\/\/yourblog.com\/blog-post\/another-post\n \u2713 Saved to: blog_posts\/another-post.md\n\n[3\/3]\nProcessing: https:\/\/yourblog.com\/blog-post\/third-post\n \u2713 Saved to: blog_posts\/third-post.md\n\n\u2713 Created index file: blog_posts\/INDEX.md\n\n============================================================\nConversion Complete!\n============================================================\nTotal posts: 3\nSuccessful: 3\nFailed: 0\nTime elapsed: 4.52 seconds\nOutput directory: \/Users\/you\/blog_converter\/blog_posts<\/code><\/pre>\n\n\n\nVerify Output<\/h3>\n\n\n\n
blog_posts\/<\/code> folder. Open one .md<\/code> file:<\/p>\n\n\n\n---\ntitle: How to Calculate ESOP Taxation in India\nslug: esop-taxation-india\nurl: https:\/\/yourblog.com\/blog-post\/esop-taxation-india\ndescription: Complete guide to ESOP taxation in India...\nauthor: Jane Doe\ncategory: ESOP Management\npublished_date: November 15, 2024\nupdated_date: November 20, 2024\nfeatured_image: https:\/\/yourblog.com\/images\/esop-tax.jpg\n---\n\n## Understanding ESOP Taxation\n\nEmployee Stock Option Plans (ESOPs) are taxed at two stages...\n\n[rest of content]<\/code><\/pre>\n\n\n\n\n
\n
<div><\/code>, <p><\/code>)<\/li>\n\n\n\nRun Full Conversion<\/h3>\n\n\n\n
python3 blog_to_markdown.py --sitemap https:\/\/yourblog.com\/sitemap.xml<\/code><\/pre>\n\n\n\nStep 6: Combine Posts for LLM Analysis<\/h2>\n\n\n\n
combine_posts.py<\/code>:<\/p>\n\n\n\n#!\/usr\/bin\/env python3\n\"\"\"\nSmart Blog Post Combiner\nCombines all markdown files into one organized document with TOC\n\"\"\"\n\nfrom pathlib import Path\nimport yaml\nimport re\nfrom datetime import datetime\n\ndef extract_frontmatter(content):\n \"\"\"Extract YAML frontmatter from markdown\"\"\"\n if content.startswith('---'):\n parts = content.split('---', 2)\n if len(parts) >= 3:\n try:\n frontmatter = yaml.safe_load(parts[1])\n body = parts[2].strip()\n return frontmatter, body\n except:\n return {}, content\n return {}, content\n\ndef sanitize_title(title):\n \"\"\"Clean title for TOC links\"\"\"\n sanitized = title.lower()\n sanitized = re.sub(r'[^\\w\\s-]', '', sanitized)\n sanitized = re.sub(r'[-\\s]+', '-', sanitized)\n return sanitized\n\ndef combine_posts(input_dir='blog_posts', output_file='all_posts_combined.md'):\n \"\"\"Combine all posts into one smart document\"\"\"\n\n input_path = Path(input_dir)\n\n # Get all markdown files except INDEX.md\n md_files = [f for f in input_path.glob('*.md') if f.name != 'INDEX.md']\n\n if not md_files:\n print(f\"No markdown files found in {input_dir}\/\")\n return\n\n print(f\"Found {len(md_files)} blog posts\")\n print(\"Reading and organizing posts...\")\n\n # Read all posts and extract metadata\n posts = []\n for md_file in md_files:\n content = md_file.read_text(encoding='utf-8')\n metadata, body = extract_frontmatter(content)\n\n posts.append({\n 'filename': md_file.name,\n 'metadata': metadata,\n 'body': body,\n 'title': metadata.get('title', md_file.stem)\n })\n\n # Sort posts by published date (most recent first)\n def get_date(post):\n date_str = post['metadata'].get('published_date', '')\n try:\n for fmt in ['%B %d, %Y', '%b %d, %Y', '%Y-%m-%d']:\n try:\n return datetime.strptime(date_str, fmt)\n except:\n continue\n except:\n pass\n return datetime.min\n\n posts.sort(key=get_date, reverse=True)\n\n print(\"Creating combined document...\")\n\n # Create the combined document\n output = []\n\n # Header\n output.append(\"# Complete Blog Collection\\n\")\n output.append(f\"**Total Posts:** {len(posts)} \")\n output.append(f\"**Generated:** {datetime.now().strftime('%B %d, %Y at %H:%M:%S')}\\n\")\n output.append(\"---\\n\")\n\n # Table of Contents\n output.append(\"## \ud83d\udcd1 Table of Contents\\n\")\n\n # Group by category\n by_category = {}\n no_category = []\n\n for i, post in enumerate(posts, 1):\n category = post['metadata'].get('category', '')\n if category:\n if category not in by_category:\n by_category[category] = []\n by_category[category].append((i, post))\n else:\n no_category.append((i, post))\n\n # Write TOC by category\n for category in sorted(by_category.keys()):\n output.append(f\"\\n### {category}\\n\")\n for idx, post in by_category[category]:\n title = post['title']\n anchor = sanitize_title(title)\n output.append(f\"{idx}. [{title}](#{anchor})\\n\")\n\n if no_category:\n output.append(f\"\\n### Other Posts\\n\")\n for idx, post in no_category:\n title = post['title']\n anchor = sanitize_title(title)\n output.append(f\"{idx}. [{title}](#{anchor})\\n\")\n\n output.append(\"\\n---\\n\")\n output.append(\"\\n# \ud83d\udcdd Blog Posts\\n\")\n\n # Add each post with clear separators\n for i, post in enumerate(posts, 1):\n metadata = post['metadata']\n title = post['title']\n\n # Post separator\n output.append(f\"\\n\\n{'='*80}\\n\")\n output.append(f\"## Post #{i}: {title}\\n\")\n output.append(f\"{'='*80}\\n\\n\")\n\n # Metadata box\n output.append(\"**Metadata:**\\n\")\n output.append(\"```yaml\\n\")\n\n meta_items = []\n if metadata.get('slug'):\n meta_items.append(f\"Slug: {metadata['slug']}\")\n if metadata.get('author'):\n meta_items.append(f\"Author: {metadata['author']}\")\n if metadata.get('category'):\n meta_items.append(f\"Category: {metadata['category']}\")\n if metadata.get('published_date'):\n meta_items.append(f\"Published: {metadata['published_date']}\")\n if metadata.get('updated_date'):\n meta_items.append(f\"Updated: {metadata['updated_date']}\")\n if metadata.get('url'):\n meta_items.append(f\"URL: {metadata['url']}\")\n\n output.append('\\n'.join(meta_items))\n output.append(\"\\n```\\n\\n\")\n\n if metadata.get('description'):\n output.append(f\"**Summary:** {metadata['description']}\\n\\n\")\n\n output.append(\"---\\n\\n\")\n\n # Post content\n output.append(post['body'])\n output.append(\"\\n\\n\")\n\n # Footer\n output.append(\"\\n\\n\")\n output.append(\"=\"*80 + \"\\n\")\n output.append(f\"**End of Collection** - {len(posts)} posts total\\n\")\n output.append(\"=\"*80 + \"\\n\")\n\n # Write to file\n output_path = Path(output_file)\n output_path.write_text(''.join(output), encoding='utf-8')\n\n size_mb = output_path.stat().st_size \/ (1024 * 1024)\n\n print(f\"\\n\u2713 Successfully combined {len(posts)} posts!\")\n print(f\"\u2713 Output file: {output_path.absolute()}\")\n print(f\"\u2713 File size: {size_mb:.2f} MB\")\n\nif __name__ == '__main__':\n import argparse\n\n parser = argparse.ArgumentParser(description='Combine blog posts')\n parser.add_argument('--input', default='blog_posts',\n help='Input directory')\n parser.add_argument('--output', default='all_posts_combined.md',\n help='Output filename')\n\n args = parser.parse_args()\n\n combine_posts(args.input, args.output)<\/code><\/pre>\n\n\n\nRun the Combiner<\/h3>\n\n\n\n
python3 combine_posts.py<\/code><\/pre>\n\n\n\nall_posts_combined.md<\/code> with:<\/p>\n\n\n\n\n
\n\n\n\nAdvanced: Scheduling Automated Updates<\/h2>\n\n\n\n
Cron Job (Mac\/Linux)<\/h3>\n\n\n\n
# Edit crontab\ncrontab -e\n\n# Add line to run weekly on Sundays at 2 AM\n0 2 * * 0 cd \/path\/to\/blog_converter && python3 blog_to_markdown.py --sitemap https:\/\/yourblog.com\/sitemap.xml<\/code><\/pre>\n\n\n\nGitHub Action<\/h3>\n\n\n\n
.github\/workflows\/blog-sync.yml<\/code>:<\/p>\n\n\n\nname: Sync Blog Posts\n\non:\n schedule:\n - cron: '0 2 * * 0' # Weekly on Sundays\n workflow_dispatch: # Manual trigger\n\njobs:\n sync:\n runs-on: ubuntu-latest\n steps:\n - uses: actions\/checkout@v3\n\n - name: Set up Python\n uses: actions\/setup-python@v4\n with:\n python-version: '3.9'\n\n - name: Install dependencies\n run: pip install -r requirements.txt\n\n - name: Run converter\n run: python blog_to_markdown.py --sitemap https:\/\/yourblog.com\/sitemap.xml\n\n - name: Commit changes\n run: |\n git config --local user.email \"action@github.com\"\n git config --local user.name \"GitHub Action\"\n git add blog_posts\/\n git commit -m \"Auto-sync blog posts\" || echo \"No changes\"\n git push<\/code><\/pre>\n\n\n\nTroubleshooting<\/h2>\n\n\n\n
“No content found” errors<\/h3>\n\n\n\n
\n
extract_content()<\/code> method with correct selector<\/li>\n<\/ol>\n\n\n\n“Command not found: python”<\/h3>\n\n\n\n
python3<\/code> instead of python<\/code><\/p>\n\n\n\n“403 Forbidden” errors<\/h3>\n\n\n\n
\n
Metadata fields are empty<\/h3>\n\n\n\n
extract_metadata()<\/code> to match your blog’s structure<\/p>\n\n\n\nBest Practices<\/h2>\n\n\n\n
1. Test Before Full Run<\/h3>\n\n\n\n
--limit 3<\/code> first to verify selectors work correctly.<\/p>\n\n\n\n2. Respect Rate Limits<\/h3>\n\n\n\n
3. Version Control Your Output<\/h3>\n\n\n\n
git init\ngit add blog_posts\/\ngit commit -m \"Initial blog export\"<\/code><\/pre>\n\n\n\n4. Document Your Selectors<\/h3>\n\n\n\n
\"\"\"\nBLOG-SPECIFIC SELECTORS (Updated: 2024-11-26)\n- Content: div.text-rich-text\n- Title: h1.heading-style-h2\n- Author: p.text-size-regular.text-color-brandnile\n- Date: p.text-size-regular.text-color-grey\n\"\"\"<\/code><\/pre>\n\n\n\n5. Regular Audits<\/h3>\n\n\n\n
\n
Extending the Script<\/h2>\n\n\n\n
Add Reading Time Calculation<\/h3>\n\n\n\n
def calculate_reading_time(content):\n \"\"\"Calculate estimated reading time\"\"\"\n words = len(content.split())\n minutes = round(words \/ 200) # Average reading speed\n return f\"{minutes} min read\"<\/code><\/pre>\n\n\n\nExtract Internal Links<\/h3>\n\n\n\n
def extract_internal_links(soup, base_url):\n \"\"\"Find all internal links for link graph analysis\"\"\"\n links = []\n for a in soup.find_all('a', href=True):\n href = a['href']\n if base_url in href:\n links.append({\n 'text': a.get_text(strip=True),\n 'url': href\n })\n return links<\/code><\/pre>\n\n\n\nGenerate Content Calendar<\/h3>\n\n\n\n
def analyze_publishing_frequency(posts):\n \"\"\"Analyze posting patterns\"\"\"\n dates = [p['metadata'].get('published_date') for p in posts]\n # Parse dates and calculate frequency\n # Suggest optimal posting schedule<\/code><\/pre>\n\n\n\nConclusion<\/h2>\n\n\n\n
\n