Create a Language of The Month answer scraper

Question

When I was posting answers for the Husk Language of The Month thread, I realized it took more time to enter them in the thread than it took to actually write the answers. So, why not automate it?

Write a program/function which inputs a language name, and displays all answers in that language during the current month so far)

Sandbox

Introduction

You are to find all answers in Code Golf Stack Exchange in the current month so far, whose first non-empty line matches the following:

#<whitespace>[<language name>(case insensitive match)]<anything else>

For example, the current month's answers should have been posted within the time frame of Nov 1 till the current date(Nov 6, as of this question's sandbox post).

for each answer, you must find the following:

• Question name
• Answer URL
• Answer Poster's Name
• Answer Poster's Profile URL

And format them in the following manner for Markdown:

- [<Question Name>](Answer URL) by [<Answer Poster's Name>]

Once you are done listing the answers, to keep the post short, you must key in the usernames to their respective profile URLs as follows:

- [<Answer Poster's Name>]:<Answer Poster's Profile URL>

This must be added at the end of the output.

Output

The final result should look like the following:

- [Size up a number](http://example1.com) by [Razetime]
- [My smartphone's Phonebook](http://example2.com) by [Bubbler]
- [1,2,Fizz,4,Buzz](http://example3.com) by [Lyxal]
- [Upgoat or Downgoat?](http://example3.com) by [LegionMammal978]
- .....

[Razetime]:http://exampleA.com
[Bubbler]:http://exampleB.com
[Lyxal]:http://exampleC.com
[LegionMammal978]:http://exampleD.com
...

Scoring

This is code-golf. Shortest answer in each language wins.

Answer 1

05AB1E, 283 282 bytes

žgŽ7?-©ƵÇ+400÷®<т÷(®69-4÷®70-365*•ΘÏF•ºS₂+žf<£`žgт‰0Kθ4ÖOŽª+·*’¸¸.‚‹º.ŒŒ/–«?€¼=ƒËŠˆ&€š‚‚=ÿ’žYì.w„†ä_¡¦ε',¡нþ}Ùε’ƒËŠˆ.‚‹º.ŒŒ/’žYì©’ÿ…é/ÿ’.w……<h1¡¦ε„><S¡3èy’="/a/’¡1è'"¡н®"ÿa/ÿ"y'†§™’ÜÇ.µÔ/ÿ’¡1è…><"S¡64Sè`®¨ì)}ćнδª}€`ʒнlIlQ}ʒ1è'/¡4èXså]DεƵ=Sè`“- [ÿ](ÿ)€‹ [ÿ]“}»sε32Sè`"[ÿ]:ÿ"}Ù»¶2×ý

Ugh..

No TIO of the full program, because .w is disabled.

Explanation:

Step 1: Get the current year and month, and convert it to a Unix timestamp (using this formula):

žg                     # Push the current year
  Ž7?-                 # Subtract 1900
      ©                # Store it in variable `®` (without popping)
       ƵÇ+             # Add 299
          400÷         # Integer-divide by 400
      ®                # Push year-1900 again from variable `®`
       <               # Subtract 1
        т÷             # Integer-divide by 100
          (            # Negate it
      ®                # Push year-1900 again
       69-             # Subtract 69
          D            # Duplicate it
           4÷          # Integer-divide the copy by 4
          s            # Swap so year-1969 is at the top again
           <           # Subtract 1 to make it year-1970
            365*       # Multiply by 365
•ΘÏF•                  # Push compressed integer 5254545
     º                 # Mirror it to 52545455454525
      S                # Convert it to a list of digits
       ₂+              # Add 26 to each: [31,28,31,30,31,30,31,31,30,31,30,31,28,31]
         žf            # Push the current month
           <           # Subtract it by 1
            £          # Leave that many trailing items in the list
             `         # Pop and push them all separated to the stack
žg                     # Push the current year again
  т‰                   # Divmod by 100: [year//100,year%100]
    0K                 # Remove 0s from this pair
      θ                # Leave the last item
       4Ö              # Check if it's divisible by 4
                       # (this results in 1 for leap years; 0 for non-leap years)
O                      # Sum all values on the stack
 Žª+                   # Push compressed integer 43200
    ·                  # Double it to 86400
     *                 # Multiply the sum by this 86400

Try just this first step online.

Step 2: Use this timestamp to create the stackexchange-API URL, and get the content of this url (which contains the JSON of all answers created this month):

’¸¸.‚‹º.ŒŒ/–«?€¼=ƒËŠˆ&€š‚‚=ÿ’
                       # Push dictionary string "api.stackexchange.com/answers?site=codegolf&fromdate=ÿ",
                       # where the `ÿ` is automatically replaced with the timestamp
 žYì                   # Prepend constant "https://"
    .w                 # Browse to this URL, and read its contents as string

Try the first two steps online (without .w).

Step 3: Extract all unique answer-id and question-ids from this JSON:

D                      # Duplicate the JSON
 „†ä_¡                 # Split it on dictionary string "question_"
      s                # Swap so the JSON is at the top again
       „Ž»_¡           # Split it on dictionary string "answer_"
            ‚          # Pair those two lists together
ε                      # Map over the pairs:
 ¦                     #  Remove the first item
  ε                    #  Map over each inner string:
   ',¡                '#   Split it on ","
      н                #   Pop and leave just the first item
       þ               #   Pop and leave just the digits in this string
  }Ù                   #  After the inner map: uniquify the resulting ids
}                      # Close the outer map

Try just this third step with the JSON as input.

Step 4: Convert each question-id to an URL, and get its contents (which is the entire HTML page of a question):

`                      # Pop and push both lists separated to the stack
 U                     # Pop the list of answer-ids and store it in variable `X`
ε                      # Map over the question-ids:
 ’ƒËŠˆ.‚‹º.ŒŒ/’       #  Push dictionary string "codegolf.stackexchange.com/"
   žYì                 #  Prepend constant "https://"
      ©                #  Store this string in variable `®` (without popping)
       ’ÿ…é/ÿ’         #  Push dictionary string "ÿ/questions/ÿ",
                       #  where the first `ÿ` is automatically replaced with "https://codegolf.stackexchange.com/"
                       #  and the second `ÿ` is automatically replaced with the current question-id
              .w       #  Browse to this URL, and read its contents as string

Try steps 3 and 4 with JSON input (without .w).

Step 5: For each HTML-content of a question, get a list of its [language_name, answer_url, user_name, user_url, question_name] (this may contain some garbage in the result - i.e. the trailing item, or if an answer contains multiple <h1>-refs, but we'll filter those out later):

…<h1¡                  #  Split the HTML-content on "<h1"
     ¦                 #  Remove the first item
      ε                #  Map over each string:
       „><S¡           #   Split it on both ">" and "<"
            3è         #   Get the fourth item (which is the question name in the first iteration,
                       #   and language name in every other iteration)
       y               #   Push the string again
        ’="/a/’¡      "#   Split it on "="/a/"
                1è     #   Get the second item (which is the trailing answer-id URL portion)
                  '"¡ '#   Split that on '"'
                     н #   Pop and leave the first item
        ®              #   Push "https://codegolf.stackexchange.com/" from variable `®`
         "ÿa/ÿ"        #   Push dictionary string "ÿa/ÿ",
                       #   where the `ÿ` are filled again to create the answer-URL
       y               #   Push the string again
        '†§           '#   Push dictionary string "person"
           ™           #   Titlecase it to "Person"
            ’ÜÇ.µÔ/ÿ’  #   Push dictionary string "schema.org/ÿ",
                       #   where the `ÿ` is replaced with "Person"
             ¡         #   Split the string on that
              1è       #   Get the second item
        …><"S¡        "#   Split that on ">", "<", and '"'
              64Sè     #   Get the seventh and fifth items
                       #   (which are the user name and trailing user URL portion)
                  `    #   Push those separated to the stack
                   ®   #   Push "https://codegolf.stackexchange.com/" again
                    ¨  #   Remove the trailing "/"
                     ì #   Prepend it in front to create the user-URL
       )               #   Wrap all four values into a list
      }ć               #  After the map: extract the first quadruple
        н              #  Pop and push its first item (the question name)
         δ             #  Map over each remaining quadruples
          ª            #   And append the question name as trailing item to each

Try just this fifth step with the HTML as input.

Step 6: Filter this list, so it only contains answers from the earlier list, and only the language that's equal to the input:

ʒ                      #  Filter this list of quintuples by:
 н                     #   Pop and leave its first item (the language name)
  l                    #   Convert it to lowercase
   Il                  #   Get the input in lowercase as well
     Q                 #   And check that both are equal
}ʒ                     #  After the filter: filter once again:
  1è                   #   Get the second item (the answer URL)
    '/¡               '#   Split it on "/"
       4è              #   Leave its fifth item (the answer-id)
         X             #   Push the list of answer-ids from variable `X`
          så           #   And check if the current answer-id is in it
]                      # Close both the filter and outer map

Try just the sixth step with the list of quintuples of the previous step as input.

Step 7: Convert it to the output in the specified format:

 €`                    # After the map over the questions, flatten it one level down
   D                   # Duplicate this list
    ε                  # Map over the quintuplets:
     Ƶ=                #  Push compressed integer 214
       Sè              #  Leave just the third, second, and fifth items
          `            #  Pop and push them separated to the stack
      “- [ÿ](ÿ)€‹ [ÿ]“ #  Push dictionary string "- [ÿ](ÿ) by [ÿ]",
                       #  where the `ÿ` are filled again
    }»                 # After the map: join the strings by newlines
   sε                  # Swap, and map over the quintuplets again:
     32Sè              #  Leave just the fourth and third items
         `             #  Pop and push them separated to the stack
      "[ÿ]:ÿ"          #  Push string "[ÿ]:ÿ",
                       #  where the `ÿ` are filled again
    }»                 # After the map: join the strings by newlines as well
      ¶2×ý             # Then join these two strings with "\n\n" as delimiter
                       # (after which the result is output implicitly)

Try just the sixth and seventh steps with the list of quintuplets of step five as input.

See this 05AB1E tip of mine (sections How to use the dictionary? and How to compress large integers?) to understand how the dictionary strings and compressed integers work.

Answer 2

Python 3.6 and later + BeautifulSoup + requests, 681 bytes

from bs4 import*
from requests import*
from datetime import datetime as d
import json
def l(b):
 c='https://';s='.stackexchange.com';z="";y="";a=json.loads(get(c+'api'+s+'/answers?site=codegolf&fromdate='+d(d.now().year,d.now().month,1).strftime('%s')).text)
 for x in a['items']:
  k=BeautifulSoup(get(f"{c}codegolf{s}/questions/{x['question_id']}#{x['answer_id']}").text);t=k.select(f"#answer-{x['answer_id']}>div>div.answercell.post-layout--right>div.s-prose.js-post-body>h1");q=k.select('.question-hyperlink')[0];n=x['owner']['display_name']
  if t and b in t[0].text:z+=f"- [{q.text}]({c+'codegolf'+s+q['href']}) by [{n}]\n";y+=f"[{n}]:{x['owner']['link']}\n"
 print(z+"\n"+y)

Relevant function is l, where you pass the name of language b

Explanation coming soon