エピソード

  • Sponsored by Linode! pythonbytes.fm/linode

    Special guest: Greg Herrera

    YouTube live stream for viewers:

    Watch on YouTube

    Michael #1: boto type annotations

    via Michael Lernerboto3's services are created at runtimeIDEs aren't able to index its code in order to provide code completion or infer the type of these services or of the objects created by them. Type systems cannot verify themEven if it was able to do so, clients and service resources are created using a service agnostic factory method and are only identified by a string argument of that method.boto3_type_annotations defines stand in classes for the clients, service resources, paginators, and waiters provided by boto3's services.

    Example with “bare” boto3:

    Example with annotated boto3:

    Brian #2: How to have your code reviewer appreciate you

    By Michael LynchSuggested by Miłosz BednarzakActual title “How to Make Your Code Reviewer Fall in Love with You”but 🤮even has the words “your reviewer will literally fall in love with you.” literally → figuratively, pleaseTopic is important though, here are some good tips:Review your own code first“Don’t just check for mistakes — imagine reading the code for the first time. What might confuse you?”Write a clear change list description“A good change list description explains what the change achieves, at a high level, and why you’re making this change.”Narrowly scope changesSeparate functional and non-functional changesThis is tough, even for me, but important.Need to fix something, and the formatting is a nightmare and you feel you must blacken it. Do those things in two separate merge requests. Break up large change listsA ton to write about. Maybe it deserves 2-3 merges instead of 1. Respond graciously to critiquesIt can feel like a personal attack, but hopefully it’s not.Responding defensively will only make things works.

    Greg #3: REPODASH - Quality Metrics for Github repositories

    by Laurence MolloyDo you maintain a project codebase on Github?Would you like to be able to show the maturity of your project at a glance?Walk through the metrics availableUse-case

    Michael #4: Extra, extra, extra, extra, hear all about it

    Python 3 Float Security BugBuilding Python 3 from source now :-/ It’s still Python 3.8.5 on Ubuntu with the kernel patch just today! (Linux 5.4.0-66 / Ubuntu 20.04.2)Finally, I’m Dockering on my M1 mac via:docker context create remotedocker --docker "host=ssh://user@server"docker context use remotedocker

    docker run -it ubuntu:latest bash now works as usual but remotely!

    Why I keep complaining about merge thing on dependabot. Why!?! ;)

    Anthony Shaw wrote a bot to help alleviate this a bit. More on that later.

    Brian #5: testcontainers-python

    Suggested by Josh PeakWhy mock a database? Spin up a live one in a docker container. “Python port for testcontainers-java that allows using docker containers for functional and integration testing. Testcontainers-python provides capabilities to spin up docker containers (such as a database, Selenium web browser, or any other container) for testing.”import sqlalchemyfrom testcontainers.mysql import MySqlContainerwith MySqlContainer('mysql:5.7.32') as mysql: engine = sqlalchemy.create_engine(mysql.get_connection_url()) version, = engine.execute("select version()").fetchone() print(version) # 5.7.32The snippet above will spin up a MySql database in a container. The get_connection_url() convenience method returns a sqlalchemy compatible url we use to connect to the database and retrieve the database version.

    Greg #6: The Python Ecosystem is relentlessly improving price-performance every day

    Python is reaching top-of-mind for more and more business decision-makers because their technology teams are delivering solutions to the business with unprecedented price-performance. The business impact keeps getting better and better. What seems like heavy adoption throughout the economy is still a relatively small-inroad compared to what we’ll see in the future. It’s like water rapidly collecting behind a weak dam.It’s an exciting time to be in the Python world!

    Extras:

    Brian:

    Firefox 86 enhances cookie protectionsites can save cookies. but can’t share between sites. Firefox maintains separate cookie storage for each site.Momentary exceptions allowed for some non-tracking cross-site cookie uses, such as popular third party login providers.

    Joke:

    56 Funny Code Comments That People Actually Wrote: These are actually in a code base somewhere (a sampling):

    /* * Dear Maintainer * * Once you are done trying to ‘optimize’ this routine, * and you have realized what a terrible mistake that was, * please increment the following counter as a warning * to the next guy. * * total_hours_wasted_here = 73 */ // sometimes I believe compiler ignores all my comments // drunk, fix later // Magic. Do not touch. /*** Always returns true ***/ public boolean isAvailable() { return false; }
  • Sponsored by Datadog: pythonbytes.fm/datadog

    Special guest: Brett Cannon

    Watch on YouTube

    Brian #1: Keeping up with Rich

    Will McGugan has been building RichIt looks like it’s on its way to becoming a full fledged TUI (text user interface)December: Live view: no blog post on that, I don’t think.January: Tree view: Rendering a tree view in the terminal with Python and RichFebruary: Layouts: Building Rich terminal dashboardsfun fullscreen.py example, uses Live viewAlso, python -m rich will display a demo screen that shows tons of the stuff that Rich can doMany of the features also have a stand alone demo built in, like: $ python -m rich.layout $ python -m rich.tree $ python -m rich.liveAlthough I haven’t figured out how to kill the live demo. it doesn’t seem to time out, and it eats Ctrl-C in my terminal.I’d really like to use Rich for interactive stuff, like keyboard interrupts and arrow keys and tab and such. It’d be fun.Which brings me to the bottom right corner of the python -m rich output. It includes a GitHub Sponsor link for Will. Also, Will, unless it’s a contradiction to RTD TOS, I think you should include a Sponsor link in the Rich documentation. Let’s convince Will to make Rich a full TUI.

    Michael #2: 12 requests per second

    If you take a look around the blogosphere at various benchmarks for Python web frameworks, you might start to feel pretty bad about your own setup.The incredible work of the guys at magic stack, getting 100,000 requests per second from uvloop in a single thread.There’s the FastAPI benchmarksEven more mind-blowing is Japronto which claims an insane 1.2 million requests per-second in a single threadBut what about your “boring” Flask or Django app? And how realistic are these benchmarks? Usually, not very.Here’s an article diving into this for a “proper” ORM style app.12 - 80 requests per sec: Both our sync workers are now hitting a blazing 12 requests per second 😅 Using async workers seems to help a lot, but oddly Sanic struggles here.Be sure to run in prod on a “real” server setup (nginx + gunicorn or whatever)Compare this to Talk Python Training - Python 3, uWSGI, Pyramid, MongoDB, $20/mo serverGet around 125 requests/sec @ 100ms response time on a single server.More realistically, we can handle about 10,000-20,000 concurrent “realistic” users before performance suffers.

    Brett #3: Python Launcher for Unix reaches RC (probably 😉)

    Exclusive! 😁Started right after PyCon US 2018Implemented in Rust (it’s my “good size” Rust learning project)The Python Launcher for Windows works by:Checking the shebang lineIf VIRTUAL_ENV is setFind the newest pythonX.Y version/command on $PATHCan specify specific versions via e.g. -3 or -3.9PY_PYTHON and PY_PYTHON3 environment variables supported--list also worksCan use PYLAUNCH_DEBUG to see what the tool is doing--help covers all of thisUnix version differs in the preference of shebang versus VIRTUAL_ENV over shebangFigure on Unix you will chmod the executable bit if you truly care about the shebangI also assume at this point people will use entry points if they really want to tie something to an interpreter versionHow often do people peg their scripts to a specific Python version instead of python2 or python3?What do people think of this logic swap (hence the “probably”)?Unix bonus feature: will automatically use any virtual environment found in .venv in the working directory (and no, what directory is considered is not configurable 😁)All of this has made it py my preferred way of running Python on my machineReally useful with Starship and its Python support (does away with the big “Tip” box they have for the python_binary setting)

    Michael #4: Build a text editor with Python and curses

    [curses](https://docs.python.org/3/library/curses.html) is a library to avoid having to deal with low level issues like efficiently painting to the terminal screen and receiving user input.a barebones curses application import curses def main(stdscr): while True: k = stdscr.getkey() if k == "q": sys.exit(0) if __name__ == "__main__": curses.wrapper(main)Clear the screen with stdscr.erase()Adding text (a line of text) to the screen: stdscr.addstr(row, 0, line)The article covers interesting topics like a “view” into the file that fits the screen and a cursor you move around with arrow keys

    Brett #5: Pattern matching and accepting change in Python

    The “5-barrel foot-gun” in the room (to use Brian’s words from the last episode 😉)Usual places have people commenting from “I like this” to screaming bloody murderI think there are many “new” people to the language who don’t know Python prior to Python 3, so they don’t realize how much things used to regularly change in the languagePattern matching was very much debated publicly, so this wasn’t a secret (and I’m sorry if you didn’t have the time to participate, but Python development doesn’t even always wait for me and I’m on the SC, so …)The 2020 SC also announced publicly the possibility of this back in December with their recommendation to accept the PEP(s)Also usual comments of “why did they waste their time on that?!? Go fix packaging!” (and it’s almost always packaging 🙄)This is open source: some people wanted to put their personal time and effort into trying to get pattern matching into Python, so that’s what they didIf you want to help with Python’s packaging ecosystem, you can do so but trying to tell people what they “need” or “should” do with their time is simply rudeHistory repeats itself: every change is unwelcome unless it solves your problemPattern matching very much opens up opportunities for certain problems that were not easily possible before, e.g. parsers and compilers are classics (and hence why they are so often implemented in functional languages)I don’t think you will see this in nearly every code base like you do e.g. list comprehensionsE.g. I’m sure data scientists aren’t saying any of this since they got @, right? 😉People also claiming it isn’t Pythonic need to note that Guido helped drive thisDo you know what is Pythonic better than Guido? 😉He might not be BDFL anymore, but that doesn’t mean he still doesn’t have good design sense, i.e. if you like Python up to this point then trust Guido’s gut that this is a good thing“In Guido We Trust” (you can even get it on a mug 😉)If you use pattern matching in real-world code and have feedback to provide with enough time to consider it before b1, then please let python-dev knowE.g. there is a chance to change the meaning of _ if that is truly your biggest hang-upThis will all probably become a blog postRunning title is “The Social Contract of Open Source”These kinds of attitudes against people trying their best to make things better for folks is what led to Guido retiring from being the BDFL in the first place, me having to take a month off from open source every year, etc.Aside: more influenced by Scala than by Haskell (not sure where Michael and some other people I’ve seen online got the idea Haskell played into this)Did you know we got list comprehensions from Haskell?

    Brian #6: A Quick Intro to Structural Pattern Matching in Python

    aka the “switch” statement. I mean, the “match” statement.Also known as PEP 636, Appendix A — Quick Introcourtesy Guido van RossumThis finally helps me to get my head around simple uses of the new syntax for 3.10simple form: def http_error(status): match status: case 400: return "Bad request" case 401: return "Unauthorized" case 403: return "Forbidden" case 404: return "Not found" case 418: return "I'm a teapot" case _: # _ acts as a wildcard and never fails to match return "Something else"you can combine several matches using | case 401|403|404: return "Not allowed"Patterns can look like unpacking assignments, and can be used to bind variables: # The subject is an (x, y) tuple match point: case (0, 0): print("Origin") case (0, y): print(f"Y={y}") case (x, 0): print(f"X={x}") case (x, y): print(f"X={x}, Y={y}") case _: raise ValueError("Not a point")This is actually the tricky one. x and/or y can change values after this statement if their case is hit.More details in the article, er, appendix.It’s still going to take a while to get used to this, but this appendix is a good start.

    Joke

    true happiness
  • エピソードを見逃しましたか?

    フィードを更新するにはここをクリックしてください。

  • Sponsored by Datadog: pythonbytes.fm/datadog

    Special guest: Hannah Stepanek

    Watch on YouTube

    Michael #1: We Downloaded 10,000,000 Jupyter Notebooks From Github – This Is What We Learned

    by Alena Guzharina from JetBrainsUsed the hundreds of thousands of publicly accessible repos on GitHub to learn more about the current state of data science. I think it’s inspired by work showcased here on Talk Python.2 years ago there were 1,230,000 Jupyter Notebooks published on GitHub. By October 2020 this number had grown 8 times, and we were able to download 9,720,000 notebooks. 8x growth.Despite the rapid growth in popularity of R and Julia in recent years, Python still remains the most commonly used language for writing code in Jupyter Notebooks by an enormous margin.Python 2 went from 53% → 11% in the last two years.Interesting graphs about package usageNot all notebooks are story telling with code: 50% of notebooks contain fewer than 4 Markdown cells and more than 66 code cells.Although there are some outliers, like notebooks with more than 25,000 code lines, 95% of the notebooks contain less than 465 lines of code.

    Brian #2: pytest-pythonpath

    plugin for adding to the PYTHONPATH from the pytests.ini file before tests runMentioned briefly in episode 62 as a temporary stopgap until you set up a proper package install for your code. (cringing at my arrogance).Lots of projects are NOT packages. For example, applications.I’ve been working with more and more people to get started on testing and the first thing that often comes up is “My tests can’t see my code. Please fix.”Exampleproj/src/stuff_you_want_to_test.pyproj/tests/test_code.pyYou can’t import stuff_you_want_to_test.py from the proj/tests directory by default.The more I look at the problem, the more I appreciate the simplicity of pytest-pythonpathpytest-pythonpath does one thing I really care about:Add this to a pytest.ini file at the proj level: [pytest] python_paths = srcThat’s it. That’s all you have to do to fix the above problem.Paths relative to the directory that pytest.ini is in. Which should be a parent or grandparent of the tests directory.I really can’t think of a simpler way for people to get around this problem.

    Hannah #3: Thinking in Pandas

    Pandas dependency hierarchy (simplified):Pandas -> NumPy -> BLAS (Basic Linear Algebra Subprograms)Languages: - - Python -> C -> Assembly df["C"] = df["A"] + df["B"] A = [ 1 4 2 0 ] B = [ 3 2 5 1 ] C = [ 1 + 3 4 + 2 2 + 5 0 + 1 ]Pandas tries to get the best performance by running operations in parallel.You might think we could speed this problem up by doing something like this: Thread 1: 1 + 3 Thread 2: 4 + 2 Thread 3: 2 + 5 Thread 4: 0 + 1However, the GIL (Global Interpreter Lock) prevents us from achieving the performance improvement we are hoping for. Below is an example of a common threading problem and how a lock solves that problem.- Thread 1 total Thread 2 1 + 3 + 4 + 2 0 0 + 5 10 0 + 6 + 2 total += 10 0 13 total =10 0 total += 13 10 total = 13 13 Thread 1 total Thread 2 1 + 3 + 4 + 2 0 unlocked 0 + 5 10 0 unlocked + 6 + 2 total += 10 0 locked 13 total =10 0 locked 10 unlocked 10 locked total += 13 10 locked total = 13 23 unlockedAs it turns out, because Python manages memory for you every object in Python would be subject to these kinds of threading issues: a = 1 # reference count = 1 b = a # reference count = 2 del(b) # reference count = 1 del(a) # reference count = 0So, the GIL was invented to avoid this headache which only lets one thread run at a time.Certain parts of the Pandas dependency hierarchy are not subject to the GIL (simplified):Pandas -> NumPy -> BLAS (Basic Linear Algebra Subprograms)GIL -> no GIL -> hardware optimizationsSo we can get around the GIL in C land but what kind of optimizations does BLAS provide us with?Parallel operations inside the CPU via Vector registersA vector register is like a regular register but instead of holding one value it can hold multiple values.| 1 | 4 | 2 | 0 | + + + +| 3 | 2 | 5 | 1 | = = = = | 4 | 6 | 7 | 1 |Vector registers are only so large though, so the Dataframe is broken up into chunks and the vector operations are performed on each chunk.

    Michael #4: Quickle

    Fast. Benchmarks show it’s among the fastest serialization methods for Python.Safe. Unlike pickle, deserializing a user provided message doesn’t allow for arbitrary code execution.Flexible. Unlike msgpack or json, Quickle natively supports a wide range of Python builtin types.Versioning. Quickle supports “schema evolution”. Messages can be sent between clients with different schemas without error.Example >>> import quickle >>> data = quickle.dumps({"hello": "world"}) >>> quickle.loads(data) {'hello': 'world'}

    Brian #5: what(), why(), where(), explain(), more() from friendly-traceback console

    Do this: $ pip install friendly-friendly_traceback.install() $ python -i >>> import friendly_traceback >>> friendly_traceback.start_console() >>>Now, after an exception happens, you can ask questions about it. >>> pass = 1 Traceback (most recent call last): File "[HTML_REMOVED]", line 1 pass = 1 ^ SyntaxError: invalid syntax >>> what() SyntaxError: invalid syntax A `SyntaxError` occurs when Python cannot understand your code. >>> why() You were trying to assign a value to the Python keyword `pass`. This is not allowed. >>> where() Python could not understand the code in the file '[HTML_REMOVED]' beyond the location indicated by --> and ^. -->1: pass = 1 ^Cool for teaching or learning.

    Hannah #6: Bandit

    Bandit is a static analysis security tool.It’s like a linter but for security issues. pip install bandit bandit -r .I prefer to run it in a git pre-commit hook:# .pre-commit-config.yaml repos: repo: https://github.com/PyCQA/bandit rev: '1.7.6' hooks: - id: banditIt finds issues like:flask_debug_truerequest_with_no_cert_validationYou can ignore certain issues just like any other linter: assert len(foo) == 1 # nosec

    Extras:

    Brian:

    Meetups this week 2/3 done.NOAA Tuesday, Aberdeen this morning - “pytest Fixtures”PDX West tomorrow - Michael Presenting “Python Memory Deep Dive”Updated my training page, testandcode.com/trainingFeedback welcome. I really like working directly with teams and now that trainings can be virtual, a couple half days is super easy to do.

    Michael:

    PEP 634 -- Structural Pattern Matching: Specification accepted in 3.10PyCon registration openPython Web Conf reg openHour of code - minecraft

    Joke:

    Sent in via Michel Rogers-Vallée, Dan Bader, and Allan Mcelroy. :)

    PEP 8 Song

    Watch on YouTube

    By Leon Sandoy and team at Python Discord
  • Sponsored by us! Support our work through:

    Our courses at Talk Python Trainingpytest bookPatreon Supporters

    Special guest: Jennifer Stark - @_JAStark & guest on talkpython.fm/259

    Watch on YouTube

    Brian #1: Do you really need a virtualenv?

    Frost Ming doesn’t think so, based on the article You don't really need a virtualenvThe link slug is “introducing-pdm”, which I think would be a better title, but the first did work to get people to talk about it. Also, “Try PEP 582 today” may have been appropriate.Teaching new people is a problem:Telling them to first type python -m venv venvThen type source venv/bin/activate or . venv/bin/activateUnless you’re on windows, then type venv\scripts\activate.batThen type pip install -r requirements.txtYeah. It’s not pretty, not fun, and good luck not having anyone ask questions about why this is necessary.Also the Python version is specified in the venv. So if you upgrade Python versions, what happens to existing venvs? The article also discusses levels of venvs, and global tools that maybe you want not tied to each venv. But we have pipx for that, so I don’t think that’s a real issue.Enter PEP 582, still in draft mode.Instead of a venv directory, your project has a __pypackage__ directory. If you python -m pip install in your project directory, stuff just goes there instead of to the global Python.So it kinda acts like a venv for local packages, it just doesn’t include local copies of the Python executables, and such.This is probably a horrible description of 582, but oh well. Something like that.pdm supports 582 todayPDM stands for Python Development Master“It installs and manages packages in a similar way to npm that doesn't need to create a virtualenv at all!”Has a workflow that reminds me of Poetry, but doesn’t use a venv, uses a package directory instead.Conclusion:Huge props to Frost for this. It’s cool to see a tool that supports 582 and glimpse a possible Python future.However, this doesn’t solve the “teaching Python” problem. The setup is more complex than venv.I’m personally sticking with venv, well virtualenv, until (and if) 582 is supported by Python and pip.

    Michael #2: Copier - like cookiecutter

    A library for rendering project templates.Works with local paths and git URLs.Your project can include any file and Copier can dynamically replace values in any kind of text file.It generates a beautiful output and takes care of not overwrite existing files unless instructed to do so.To use as a CLI app: pipx install copierTo use as a library: pip install copierHas a simple Python APIMain advantage: Can update existing projectsRuns from basic YAML files

    Jennifer #3: Pandarallel - run pandas apply in parallel!

    simple install `pip install pandarallel [--upgrade] [--user]``import from pandarallel import pandarallelinitialise pandarallel.initialize(), set progress bar BOOL, set number of workers … (defaults to all cores)just use parallel_apply where you’d usually put apply

    Brian #4: Stop Using Print to Debug in Python. Use icecream Instead

    Khuyen Tranprint(f``"``{x=}``"``) is better than print(f``"``x: {x}``"``) but it’s still a lot of typing.With icecream, you can type ic(x) insted and git this nice output: ic| x: 5It’s less typing and just as nice. There’s more.ic() with no arguments logs the file, function, line number when it’s hit. Easy program flow tracing without a debugger.You can configure it to do this cool context thing even if you do pass in a value to print.You can configure custom prefix formatting with a callback function, so you can include the time or the user that’s logged in, or whatever else state you want to track.Since all output is prefixed with ic|, you can see it easilyWrites to stderr by default, so it doesn’t muck up stdout stuffClean it out of your code by searching for ic() statements. If you have normal print statements in your code, you don’t want to use print for debugging also.

    Michael #5: HTMX: Dynamic and live HTML without JavaScript

    htmx allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTMBest seen via the examples section - try some out live on their siteHas a cool Server Requests pane for seeing what’s happening in the example

    Jennifer #6: PyLDAvis - Interactive Topic Model Visualisation

    Port of LDAvis R package (does this mean PyLDAvis is a wrapper? A translation?) by Carson Sievert and Kenny ShirleyUser calls pyLDAvis with fitted model made with your favourite library (eg Gensim, sklearn, GraphLab)Outputs include:term frequency within topic bar chartterm frequency within whole corpus bar chartnext to each bar is a word. You hover over the word and the topic circles adjust size to reflect representation of that term in that topic. topic circles - one for each topic, whose areas are setto be proportional to the proportions of the topics across the N total tokens in the corpusterm-topic circles, with area proportional to the frequencies with which a given term is estimated to have been generated by the topics of whole corpusslider to adjust relevance metric (0 = terms very specific to currently selected topic; 1 = terms frequently seen in many topics).

    Extras:

    Brian:

    I’m also speaking to a group of NOAA people next week.I’m speaking the Aberdeen Python User Group on the 10th of Feb. It’s virtual, so everyone can come.Excited about both. My kids are more impressed with the NOAA thing. It’s fun to impress your kids.

    Michael:

    Jet Brain’s fifth annual Developer Ecosystem survey

    Joke:

    Engineer helping a designer

    https://twitter.com/EduardoOrochena/status/1306944019268861953

  • Sponsored by Datadog: pythonbytes.fm/datadog

    Special guest: Jeremy Tanner

    Watch on YouTube

    Brian #1: Constant Folding in Python

    Arpit BhayaniConstant Folding is when a language replaces constant expressions at compile time rather than computing them at runtime.CPython does this while creating the bytecode.We can use dis to see it in action >>> import dis >>> dis.dis("day_sec = 24 * 60 * 60") 1 0 LOAD_CONST 0 (86400) 2 STORE_NAME 0 (day_sec) 4 LOAD_CONST 1 (None) 6 RETURN_VALUEPython tries to fold lots of constants, but not all.Seems to be based on size >>> x = 2 ** 64 # folded >>> x = 4 ** 64 # not folded >>> a = "-" * 4096 # folded >>> a = "-" * 4097 # not foldedDiscussion continues with a discussion of how CPython folding is implemented recursively and elegantly.Key takeaway for me: Remember to not try to pre-compute constants while coding. Make them easy to read and Python will handle the optimization for you.

    Michael #2: Update All Packages With pip-review

    via PyCodersUpdating Python packages can be a hassle.There are many of them - it's hard to keep track of all the newest versionsWhen you decide what to update, you still have to update each of them manually.Originally a part of the pip-tools package, it now lives on as a standalone convenience wrapper around pip.Usage (venv) $ pip install pip-review (venv)$ pip-review scikit-learn==0.23.2 is available (you have 0.23.1) scipy==1.5.4 is available (you have 1.4.1) seaborn==0.11.0 is available (you have 0.10.1) ...Once you've identified if you'd like to update your packages, you can update them all, automatically, using pip-review --autoLimit with constraints.txt fileCan do it interactively via pip-review --interactive

    Jeremy #3: Quantum Mechanical Keyboard Firmware

    How does the Python get into your computer? Much of it, through the keyboardWhy mechanical keyboards?ErgonomicsWhere is QWERTY from?And when?Keymapsand because it’s essentially an escape room built of hardware and open source software and dodgy international transactionsI’ve fallen further down the mechanical keyboard rabbit holeRGBKB SolKeyboardio AtreusKeyboard firmware is mostly written in chttps://github.com/qmkBut there are Python portionshttps://github.com/qmk/qmk_cliCompiler helperAPIReportingSo the CLI is Python, where else is Python?When looking to improve your keymap, key logging w/ Python to process logs for heatmapshttps://github.com/qmk/qmk_firmware/issues/714https://github.com/algernon/ergodox-layout/blob/master/tools/log-to-heatmap.py

    Brian #4: Reinventing the Python Logo

    Carlo Occhiena interview with Jessica WilliamsonSome cool logo history

    Early logo

    Current, from 2006, designed by Tim Parklin

    “The logo is actually based on mayan representations of snakes which very often represent only the head and perhaps a short length of tail. The structure of the snake representations the natural coiling/nesting of a snake as seen side on.” - Tim Parklin

    Jessica Williamson proposed a new one in 2020:

    Then there’s the rest of the article talking about the design process, etc.

    But….. just wanted to throw this out there. I’m happy with the 2006 version. - BrianMK: Have you ever seen the logos on the app stores?

    Michael #5: Private PyPI with Serverless Computing

    Project: aws-lambda-pypiBrings together a few pieces so you can easily run a small PyPICloud instance as an AWS Lambda function with the help of Terraform.PyPICloud lets you publish Python packages privately.AWS Lambda lets you run a small service for free.Terraform ensures the service is deployed and maintained correctly.Security: This project takes the following security precautions.The session keys are auto-generated and stored in a secret.The server configuration file, which contains the session keys, is generated on the fly to minimize the possibility of exposure.The lambda function runs with a role limited to accessing only what it needs.PyPICloud is configured to display no packages to unauthenticated users.

    Jeremy #6: Beyond the Basic Stuff w/Python

    Al SweigartWant to become less feral?

    Extras:

    Jeremy:

    Twttier Haiku bot: @hitch_haiku

    Michael:

    Django 3.2 alpha 1 releasedmypy 0.800 with Python 3.9 Supportpip 21.0 out and it drops Python 2Elastic changes open-source license to monetize cloud-service use

    Brian:

    We talked about pip-chill in episode 208 pip-chill now has a --no-chill option to not list itself. Nice.I was just on https://www.twitch.tv/microsoftdeveloper, short, but fun

    Joke

    via Wolf

    by Kat Maddox

    developer: so i have good news and bad news

    manager: what's the good news?developer: i've discovered that the "5 second rule" only applies to foodmanager: and the bad news?developer: i dropped our tables
  • Sponsored by Linode! pythonbytes.fm/linode

    Special guest: Ogi Moore

    Watch the live stream on YouTube.

    Michael #1: diskcache

    via Ian MaurerPython disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.The cloud-based computing of 2020 puts a premium on memory. Gigabytes of empty space is left on disks as processes vie for memory. Among these processes is Memcached (and sometimes Redis) which is used as a cache. Wouldn't it be nice to leverage empty disk space for caching?Features:Pure-PythonFully DocumentedBenchmark comparisons (alternatives, Django cache backends)100% test coverageHours of stress testingPerformance mattersDjango compatible APIThread-safe and process-safeSupports multiple eviction policies (LRU and LFU included)Keys support "tag" metadata and evictionDeveloped on Python 3.8Tested on CPython 3.5, 3.6, 3.7, 3.8Tested on Linux, Mac OS X, and WindowsTested using Travis CI and AppVeyor CI

    Brian #2: TOML is 1.0.0 now.

    What does that mean for Python?Hopefully, some kind of toml parser will make it into Python core. Any Python access to 1.0.0? MixedImplementations and TOML version support page lists:pytomlpp supports 1.0.0-rc.3, which is a wrapper around C++ tomlplusplus, which does support 1.0.0. Confusingtomlkit supports 1.0.0-rc.1, so that’s promisingtoml supports 0.5.0, great name. It’d be cool if it would support 1.0.0What’s different between 0.5.0 and 1.0.0?Unless I’m mistaken, not much: CHANGELOG1.0.0-rc1Leading zeroes in exponent parts of floats are permitted.Allow raw tab characters in basic strings and multi-line basic strings.Allow heterogenous values in arrays.Other than that, lots of “Clarify …”, which I’m not sure how those all affect implementation.I’d love to hear more from people who know more about this

    Ogi #3: pyqtgraph

    pyqtgraph - plotting library, for when you need fast/interactive plotsUses qt5 (and soon qt6) bindings to generate plots within Qt applicationsFills a niche role, want easy mouse interactivity, running locally on a machineOften used in engineering/scientific applications when looking at a lot of data, and wanting interactivity

    Michael #4: Parler + Python = Insurrection in public

    via Jim Kring and Mark LittleAccording to Wikipedia: Parler (/ˈpɑːrlər/) is an American alt-tech microblogging and social networking service. Parler has a significant user base of Donald Trump supporters, conservatives, conspiracy theorists, and right-wing extremists.ArsTechnica article send in by Mark LittleArs: Parler’s amateur coding could come back to haunt Capitol Hill riotersCoding messA key reason for her success: Parler’s site was a mess. Its public API used no authentication. When users deleted their posts, the site failed to remove the content and instead only added a delete flag to it. Oh, and each post carried a numerical ID that was incremented from the ID of the most recently published one.Another amateur mistake was Parler’s failure to scrub geolocations from images and videos posted online.Some 80 terabytes of posts, 1M videos, many already deleted, preserved for posterity.Catalog and Python pointed out by Shaun King.See the catalog (maybe, it’s the ugly side of people).The gist: https://gist.github.com/kylemcdonald/d8884da1a82ef50754ee49e0b6561071Partially back online with Russian hosting service?

    Brian #5: Best-of Web Development with Python

    Suggested by Douglas NicholsCool list with nice iconsCoversFrameworks, HTTP Clients, ServersAuth tools, HTML Processing, URL utilitiesOpenAPI, GraphQL, WebsocketRPC, Serverless, Content ManagementWeb Testing, Web Forms, MarkdownThird-party APIsEmail, Web Scraping & Crawling, MonitoringAdmin UIAPI ProxiesFlask/FastAPI/Pyramid/Django UtilitiesNice to see lots of FastAPI projects:fastapi-sqlalchemy - Adds simple SQLAlchemy support to FastAPI. fastapi-plugins - FastAPI framework plugins. fastapi_contrib - Opinionated set of utilities on top of FastAPI. starlette_exporter - Prometheus exporter for Starlette and FastAPI. fastapi-utils - Reusable utilities for FastAPI. fastapi-code-generator - This code generator creates FastAPI app from an.. slowapi - A rate limiter for Starlette and FastAPI. fastapi-versioning - api versioning for fastapi web applications. fastapi-react - Cookiecutter Template for FastAPI + React Projects. Using.. fastapi_cache - FastAPI simple cache.

    Ogi #6: Assorted

    Pyjion - https://github.com/tonybaloney/Pyjion a JIT extension for CPython that compiles python code using .NET 5 CLR CuPy - NumPy compatible multi-dimensional array on CUDA, uses _``*array_function_* (enabled with numpy 1.17) code using numpy to operate directly on CuPy arrayssee NEP-18 and CuPy docscompatible with other libraries as well

    Extras:

    Michael:

    Trying Firefox + Brave + VPNPython Web Conf 2021 call for talks, due Jan 29, I’ll be speaking!PyCon US 2021 launched call for proposals:December 22, 2020 — Call for proposals openedFebruary 12, 2021 — Proposals are dueMarch 16, 2021 — Notifications will be sent to presentersMarch 23, 2021 — Deadline for speakers to confirm participationMarch 30, 2021 — Schedule is publicly releasedApril 28, 2021 — Deadline to submit pre-recorded presentation (tutorials will be live)May 12-13, 2021 — Tutorial daysMay 15-16, 2021 — Conference daysApple launching Racial Equity and Justice Initiatives with partners across a broad range of industries and backgrounds — from students to teachers, developers to entrepreneurs, and community organizers to justice advocates

    Brian:

    PyCascades 2021 schedule https://2021.pycascades.com/program/schedule/

    Ogi:

    Anthony Explains Video SeriesLearn X in Y minutesReading Working in Public by Nadia Eghbal - provides some sanity checks for existing maintainers, might be a fantastic perspective for new contributors to open source

    Joke

    Tech Support, 2x

    Working at the help desk? Get the theme song:

    Here to help song

    And help by chat:

    "Running a successful open source project is just Good Will Hunting in reverse, where you start out as a respected genius and end up being a janitor who gets into fights." - Byrne Hobart
  • Sponsored by Datadog: pythonbytes.fm/datadog

    Special guest: Jousef Murad, Engineered Mind podcast (audio, video)

    Watch on YouTube

    Brian #1: pip search. Just don’t.

    pip search [query] is supposed to “Search for PyPI packages whose name or summary contains [query]”The search feature looks like it’s going to be removed and the PyPI api for it removed.Alternative, and better approach, just manually look at pypi.org and search for stuff. Right now it does this: $ pip search pytest ERROR: Exception: Traceback (most recent call last): ... [longish traceback ommited] --- xmlrpc.client.Fault: [Fault -32500: "RuntimeError: PyPI's XMLRPC API has been temporarily disabled due to unmanageable load and will be deprecated in the near future. See https://status.python.org/ for more information."]The Python Infrastructure status page says, as of Jan 12: “Update - The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.”This started becoming a problem in mid December.The endpoint was just never architected to handle the scale it’s getting now. There’s a current issue “Remove the pip search command”, open on pip.The commend thread is locked now, but you can read some of the history.I personally don’t understand the need to hammer search with a CI system or other. Probably should be using a local cache or local pypi mirror for an active/aggressive CI system.If you have scripts or jobs that run pip search , it ain’t gonna work, so probably best to remove that.

    Michael #2: QPython - Scripting for Android with Python

    Python REPL on Android - interestingScripting Android tasks with Python - more interestingFree, open source app that is ad supported.Some people have commented that their phone is their only “computer”With SL4A features, you can use Python programming to control Android work:Android Apps API, such as: Application, Activity, Intent & startActivity, SendBroadcast, PackageVersion, System, Toast, Notify, Settings, Preferences, GUIAndroid Resources Manager, such as: Contact, Location, Phone, Sms, ToneGenerator, WakeLock, WifiLock, Clipboard, NetworkStatus, MediaPlayerThird App Integrations, such as: Barcode, Browser, SpeechRecongition, SendEmail, TextToSpeechHardwared Manager: Carmer, Sensor, Ringer & Media Volume, Screen Brightness, Battery, Bluetooth, SignalStrength, WebCam, Vibrate, NFC, USB

    Jousef #3: Thesis: Deep Learning assistant for designers/engineers

    PyTorch (3D) / TensorFlowThe thesis: what is it actually about & goal of the thesisLibraries mainly used: numpy, pandas(Reinforcement Learning & GANs)

    Brian #4: sortedcontainers

    Thanks to Fanchen Bao for the topic suggestion.Pure-Python, as fast as C-extensions, sorted collections library. >>> from sortedcontainers import SortedList >>> sl = SortedList(['e', 'a', 'c', 'd', 'b']) >>> sl SortedList(['a', 'b', 'c', 'd', 'e']) >>> sl *= 10_000_000 >>> sl.count('c') 10000000 >>> sl[-3:] ['e', 'e', 'e'] >>> from sortedcontainers import SortedDict >>> sd = SortedDict({'c': 3, 'a': 1, 'b': 2}) >>> sd SortedDict({'a': 1, 'b': 2, 'c': 3}) >>> sd.popitem(index=-1) ('c', 3) >>> from sortedcontainers import SortedSet >>> ss = SortedSet('abracadabra') >>> ss SortedSet(['a', 'b', 'c', 'd', 'r']) >>> ss.bisect_left('c') 2“All of the operations shown above run in faster than linear time.”Types:SortedListSortedKeyList (like SortedList, but you pass in a key function, similar to key in Pythons sorted function.)SortedDictSortedSetGreat documentation and tons of performance metrics in the docs.

    Michael #5: Łukasz Langa Typed Twitter Thread

    Let’s riff on typing for a bit. Here is my philosophy: If I have to type more than three characters to complete a symbol in my editor, something is wrong. e.g. to go from email_service. → email_service.send_account_email() I should only need to type .sae then tab/enter. These types of things are vastly better because of type hints.Python type hints are more malleable than even TypeScript.Lukasz is addressing this comment: Controversial take: Types in a Python code-base are a net negative.Pointsput enough annotations and tooling connects the dots, making plenty of errors evident.The most common to me at least is when a None creeps in. The second bug often caught by type checkers is on the "return" boundary: one of your code paths forgets a return.squiggly lines in your editorMicrosoft is now developing powerful type checking and code completion for Python in VSCode. This effort employs a member of the Python Steering Council, and possibly also the creator of Python himself soon. You think they would settle for "illusion of productivity"?

    Jousef #6:

    Point Cloud operations → open3d

    Extras:

    Michael:

    via Francisco Giordano Silva: On Brian's ref to using numpy all for array element-wise comparison, also please check out numpy.allclose method. Allows you to compare two arrays based on a given tolerance.

    Brian:

    Just this: 2021 is exhausting so far.Test & Code has shifted to every other week to allow time for other projects I’m working on.This is probably a short term change. But I don’t know for how long. It’s definitely not going away though. Just slowing down a bit.

    Jousef: Engineered Mind podcast

  • Sponsored by us! Support our work through:

    Our courses at Talk Python Trainingpytest bookPatreon Supporters

    Special guest: Jason McDonald

    Watch on YouTube

    Michael #1: 5 ways I use code as an astrophysicist

    Video by Dr. Becky (i.e. Dr Becky Smethurst, an astrophysicist at the University of Oxford)She has a great YouTube channel to check out.#1: Image Processing (of galaxies from telescopes)Noise removal#2: Data analysisImage features (brightness, etc)One example: 600k “rows” of galaxy properties#3: Model fittinge.g. linear fit (visually as well through jupyter)e.g. Galaxies and their black holes grow in mass togetherColor of galaxies & relative star formation#4: Data visualization#5: SimulationsBeautiful example of galaxies collidingStar meets black hole

    Brian #2: A Visual Intro to NumPy and Data Representation

    Jay AlammarI’ve started using numpy more frequently in my own work.Problem: I think of np.array like a Python list. But that’s not right.This visualization guide helped me think of them differently.Covers:arrayscreating arrays (I didn’t know about np.ones(), np.zeros(), or np.random.random(), so cool)array arithmeticindexing and slicingaggregation with min, max, sum, mean, prod, etc.matrices : 2D arraysmatrix arithmeticdot product (with visuals, it takes seconds to understand)matrix indexing and slicingmatrix aggregation (both all entries and column or row with axis parameter)transposing and reshapingndarray: n-dimensional arraystransforming mathematical formulas to numpy syntaxdata representationAll with excellent drawings to help visualize the concept.

    Jason #3: Qt 6 release (including PySide2)

    Qt 6.0 released on December 8: https://www.qt.io/blog/qt-6.0-released3D Graphics abstraction layer called RHI (Rendering Hardware Interface), eliminating hard dependency on OpenGL, and adding support for DirectX, Vulkan, and Metal. Uses native 3D graphics on each device by default.Property bindings: https://www.qt.io/blog/property-bindings-in-qt-6A bunch of refactoring to improve performance.QtQuick stylingCAUTION: Many Qt 5 add-ons not yet supported!! They plan to support by 6.2 (end of September 2021).Pay attention to your 5.15 deprecation warnings; those things have now been removed in 6.0.PySide6/Shiboken6 released December 10: https://www.qt.io/blog/qt-for-python-6-releasedNew minimum version is Python 3.6, supported up to 3.9.Uses properties instead of (icky) getters/setters now. (Combine with snake_case support from 5.15.2) from __feature__ import snake_case, true_propertyPyQt6 also just released, if you prefer Riverbank’s flavor. (I prefer official.)

    Michael #4: Is your GC hyper active? Tame it!

    Let’s think about gc.get_threshold().Returns (700, 10, 10) by default. That’s read roughly as:For every net 700 allocations of a collection type, a gen 0 collection runsFor every gen 0 collection run, 1/10 times it’ll be upgraded to gen 1.For every gen 1 collection run, 1/10 times it’ll be upgraded to gen 2. Aka for every 100 gen 0 it’s upgraded to gen 2.Now consider this: query = PageView.objects(created__gte=yesterday).all() data = list(query) # len(data) = 1,500That’s multiple GC runs. We’ve allocated at least 1,500 custom objects. Yet never ever will any be garbage.But we can adjust this. Observe with gc.set_debug(gc.DEBUG_STATS) and consider this ONCE at startup: # Clean up what might be garbage gc.collect(2) # Exclude current items from future GC. gc.freeze() allocs, gen1, gen2 = gc.get_threshold() allocs = 50_000 # Start the GC sequence every 10K not 700 class allocations. gc.set_threshold(allocs, gen1, gen2) print(f"GC threshold set to: {allocs:,}, {gen1}, {gen2}.")May be better, may be worse. But our pytest integration tests over at Talk Python Training run 10-12% faster and are a decent stand in for overall speed perf.Our sitemap was doing 77 GCs for a single page view (77!), now it’s 1-2.

    Brian #5: Top 10 Python libraries of 2020

    tryolabscriteriaThey were launched or popularized in 2020.They are well maintained and have been since their launch date.They are outright cool, and you should check them out.

    General interest:

    Typer : FastAPI for CLI applicationsI can’t believe first commit was right before 2020. Just about a year after the introduction of FastAPI, if you can believe it.Sebastián Ramírez is on 🔥 Rich : rich text and beautiful formatting in the terminal.Will McGuganyep. showed up in 2020, amazing.Dear PyGui : Python port of the popular Dear ImGui C++ project.PrettyErrors : transforms stack traces into color coded, well spaced, easier to read stack traces.Diagrams : lets you draw the cloud system architecture without any design tools, directly in Python code.

    Machine Learning:

    Hydra and OmegaConfPyTorch LightningHummingbirdHiPlot : plotting high dimensional data

    Also general

    Scalene : CPU and memory profiler for Python scripts capable of correctly handling multi-threaded code and distinguishing between time spent running Python vs. native code, without having to modify your code to use it.

    Jason #6: Adoption of pyproject.toml — why is this so darned controversial?

    The goal of this file is to have a single standard place for all Python tool configurations. It was introduced in PEP 518, but the community seems divided over standardizing.

    A bunch of tools are lagging behind others in implementing. Tracked in this repo

    A few of the bigger “sticking points”:

    setuptools is working on it: https://github.com/pypa/setuptools/issues/1688MyPy: GVR says it “doesn’t solve anything” and closed the PR. https://github.com/python/mypy/issues/5205Flake8 objections: https://gitlab.com/pycqa/flake8/-/issues/428#note_251982786Lack of standard TOML parser.“pip to change its behavior so mere presence of the file does not change functionality”Flake9 already implemented what Flake8 wouldn’t. Is this political?Bandit is sitting on a PR since 2019: https://github.com/PyCQA/bandit/issues/550ReadTheDocs: It’s too much work? — https://github.com/readthedocs/readthedocs.org/issues/7065PyOxidizer (shockingly), silent on the topic since 2019: https://github.com/indygreg/PyOxidizer/issues/93

    Extras:

    Michael:

    PyXLL for Excel people, including Python Jupyter Notebooks in Excel.Django 3.1.5 ReleasedPython 3.10.0a4 Is Now Available for TestingSciPy 1.6.0 ReleasedM1 + PyCharm fast? Example.Flying solo with the M1 too - apparently 75% is shutdown time for my MBP!

    Joke

    “Why did the programmer always refuse to check his code into the repository? He was afraid to commit.”

  • 2020 will be one for the history books, won't it? I've put together a great group to look back on 2020 - from the Python perspective.

    Join Brian and Michael along with Cecil Phillip, Ines Montani, Jay Miller, Paul Everitt, Reuven Lerner, and Matt Harrison for a light-hearted and fun look back on the major Python events of 2020.

    Video version of this episode: Watch on YouTube

    GuestsCecil PhillipInes MontaniJay MillerPaul EverittReuven LernerMatt Harrison
  • Sponsored by us! Support our work through:

    Our courses at Talk Python TrainingTest & Code PodcastPatreon Supporters

    Special guest: Anthony Shaw

    Stream on YouTube as wellWatch on YouTube

    Michael #1: Django Ledger Project

    by Miguel SandaThe mission is to provide free and open source accounting software that could easily replace commercial alternatives like QuickBooks.Django Ledger supports:Chart of Accounts.Financial Statements (Income Statement & Balance Sheets).Automatic financial ratio & insight calculations.Multi tenancy.Hierarchical entity management.Self-contained Ledgers, Journal Entries & Transactions.Financial Activities Support (operational/financial/investing).Basic OFX & QFX file import.Bills & Invoices with optional progressible functionality.Basic navigational templates.Entity administration & entity manager support.Bank Accounts.Not quite ready for production.This project is actively looking for contributors. Any financial and/or accounting experience is a big plus. If you have prior accounting experience and want to contribute, don't hesitate to contact.Browse the screenshots on the github repo.

    Brian #2: Flask-Meld: simple JavaScript interactive features without all of the JavaScript.

    Flask-Meld article with exampleMichael Abrahamsen, @MikeAbrahamsenworking example, code for exampleIt seems really zippy. Definitely gonna try this.Similar project for Django: django-unicornAdam Hill, @adamghilltodo example

    Anthony #3: Bitwise operators in Python (RealPython)

    Article by Bartosz ZaczyńskiParticularly useful for the enum.IntFlag and enum.Flag classes in the stdlib since 3.7 >>> class Color(Flag): ... RED = auto() ... BLUE = auto() ... GREEN = auto() ... WHITE = RED | BLUE | GREEN

    Michael #4: Why should you use an ORM (Object Relational Mapper)?

    You may have heard you should use an ORM, but why?To get a better understanding of how ORMs work, it’s helpful to work through the kind of problems they can solve.Data modeled as classes is great but has some shortcomingsHow do you query, filter, sort it?How do you store it? Classes live in memory only and pickling isn’t ideal for many reasonsThere is also concurrency, transactional operations/atomiticityEnter the DBBut plain SQL is error prone (did you refactor that name? oops)Plain SQL has code written within another language (SQL within Python, tools help but still)Plain SQL has potential security issues (unless you use parameters)ORMsORMs are libraries that sit between the SQL database world and your object-oriented programming language world.For all intents and purposes, they are an abstraction layer that allows, among other things, for the translation of Python to SQL.A cycle is formedDefine our models in the Python world as objectsthe ORM layer translates those models to SQL statements with SQL types to create tables and columns. In our application, we instantiate objects from those models, which creates rows in the aforementioned tables. Finally, when we want those objects back, we use Python code to retrieve them, which triggers the ORM layer to create the necessary query to retrieve the rows from the database. Then, the ORM layer takes the resulting rows and maps them back to objects.And you get migrations

    Brian #5: sqlite-utils: a Python library and CLI tool for building SQLite databases

    Simon Willison, @simonw“sqlite-utils is a combination Python library and command-line tool I’ve been building over the past six months which aims to make creating new SQLite databases as quick and easy as possible.”CLI for sqliteRun queries and output JSON, CSV, or tablesanalyzing tablesinserting datacreating, dropping, transforming tablescreating indexessearching…Python API for using as a libraryway easier interaction with sqlite. import sqlite_utils db = sqlite_utils.Database("demo_database.db") # This line creates a "dogs" table if one does not already exist: db["dogs"].insert_all([ {"id": 1, "age": 4, "name": "Cleo"}, {"id": 2, "age": 2, "name": "Pancakes"} ], pk="id")sqlite-utils docs

    Anthony #6: Online conferences are not working for me. But this was a good talk,

    “What the struct?!” by Zachary Anglin at the Pyjamas conference.Explains the struct libraries macro-language for converting binary data structures into Python native types (and vice versa) https://docs.python.org/3/library/struct.html#format-charactersReally nice if you want to programmatically edit/read binary structure (combining with the bitwise operators).If you’re getting into this topic. Save-game hacking is an interesting place to start. I use SynalizeIT! /Synalysis as a binary data grammar explorer. It has some example save game formats https://www.synalysis.net/formats.xmlIts also great for hacking/CTF.

    https://youtu.be/QT_NAk_peHQ?t=7130

    Extras

    Brian:

    Resurrecting Python PDX West as a virtual lunchtime event.January is planned for January 14, 2021

    Michael:

    Did a FastAPI webcast people can check out.M1 life is getting better and better

    Anthony:

    From February 1 I’ll be the new Python Open Source advocate at Microsoft, working with Nina ZacharenkoGive us a quick update on Pyjion + .NET Core?

    Jokes:

    https://twitter.com/lk012/status/1334390836172378113

  • Sponsored by us! Support our work through:

    Our courses at Talk Python TrainingTest & Code PodcastPatreon Supporters

    Special guest: Shari Eskenas

    Catch the video edition live stream on YouTubeWatch on YouTube

    Be part of the episode by subscribing and “smashing that bell” over at pythonbytes.fm/youtube

    Brian #1: pytest 6.2 is out

    pytester fixture for plugin testing. Like testdir, but a better name, and uses pathlib.Path objects.verbose mode now shows the reason a test was skipped in the terminal line after the SKIPPED, XFAIL, or XPASSCan use monkeypatch as a context manager with .context() and it’s available both in test and fixture functions, but also in helper functions by using pytest.Monkeypatch.context(). import os from contextlib import contextmanager import pytest def test_foo_1(monkeypatch): with monkeypatch.context() as mp: mp.setenv("foo", "bar") assert os.getenv("foo") == "bar" @contextmanager def some_func(): with pytest.MonkeyPatch.context() as mp: mp.setenv("foo", "bar") yield def test_foo_2(): with some_func(): assert os.getenv("foo") == "bar"Lots of other goodies.related: pytest-check, my only released plugin for pytest, was updated to support pytest 6.x

    Michael #2: SQLite as a file format (like docx)

    via Jon BultmeyerAn SQLite database file with a defined schema often makes an excellent application file format. Here are a dozen reasons why this is so:Simplified Application Development. No new code is needed for reading or writing the application file.Single-File Documents. An SQLite database is contained in a single file, which is easily copied or moved or attached.High-Level Query Language. SQLite is a complete relational database engine, which means that the application can access content using high-level queries.Accessible Content. Information held in an SQLite database file is accessible using commonly available open-source command-line tools - tools that are installed by default on Mac and Linux systems and that are freely available as a self-contained EXE file on Windows.Cross-Platform. SQLite database files are portable between 32-bit and 64-bit machines and between big-endian and little-endian architectures and between any of the various flavors of Windows and Unix-like operating systems.Atomic Transactions. Writes to an SQLite database are atomic. They either happen completely or not at all, even during system crashes or power failures.Incremental And Continuous Updates. When writing to an SQLite database file, only those parts of the file that actually change are written out to disk. This makes the writing happen faster and saves wear on SSDs.Easily Extensible. As an application grows, new features can be added to an SQLite application file format simply by adding new tables to the schema or by adding new columns to existing tables. Adding columns or tables does not change the meaning of prior queries.Performance. In many cases, an SQLite application file format will be faster than a pile-of-files format or a custom format.Concurrent Use By Multiple Processes. SQLite automatically coordinates concurrent access to the same document from multiple threads and/or processes.Multiple Programming Languages. Though SQLite is itself written in ANSI-C, interfaces exist for just about every other programming language you can think of.Better Applications. If the application file format is an SQLite database, the complete documentation for that file format consists of the database schema, with perhaps a few extra words about what each table and column represents.

    Shari #3: A Day in Code: Python – A picture book written in code

    Brian #4: PythonLabs is now hosted by Azure. and “Yes, Barry, there is a PythonLabs”

    I can’t believe we haven’t covered this in the last 211 episodes.But it seems like good timing now.Now resides at azure.pythonlabs.comBy Tim Peters (originally posted on January 6th, 2004 to the PSF members list)Barry (I’m assuming Barry Warsaw) asked the question: “… what /is/ Pythonlabs now? Or /is/ there a Pythonlabs now? I dunno -- Guido owns the domain name which is probably the biggest claim to Pythonlabhood there is.”Tim replies with a very “Yes, Virginia, there is a Santa Clause”-esque answer:

    Snippets include:

    Barry, your little friends are wrong. They have been affected by the skepticism of a skeptical age. …Yes, Barry, there is a PythonLabs. It exists as certainly as love and generosity and devotion exist, … Alas! how dreary would be the world if there were no PythonLabs! It would be as dreary as if there were no Barrys. …Not believe in PythonLabs! You might as well not believe in fairies. … Nobody sees PythonLabs, but that is no sign that there is no PythonLabs. …A thousand years from now, Barry, nay 10 times 10,000 years from now, it will continue to make glad the heart of childhood.

    Michael #5: Extra, extra, extra, extra, extra, extra, hear all about it

    #1 Numpy version pinning via GriceJust catching up on Ep 208. Note in part 2 about the Numpy issue, folks can pin versions by platform with environment markers: numpy==1.19.3; platform_system == 'Windows' numpy==1.19.4; platform_system == 'Linux'#2 Stylesheet for PySide2 and PyQt5, this time looks like Material Design - via William Jamir Silva#3 Talk Python hits 20M downloads. Python Bytes is almost 6M too.#4 Pyramid 2.0 is coming.#5 Python 3.9.1 is out with 282 changes. Ships as a universal binary (Intel + M1) on macOS.#6 Python + Mac Mini + M1 video#7 Python Steering Council selected

    Shari #6: OpenMV

    Extras

    Brian:

    agrs, kwargs (quargs), and communitya silly discussion on twitterNocole Carlson: “Huge debate … about whether you say “quargs” or “keyword arguments” for “kwargs”. Obviously “quargs” is correct.”It never occurred to me to say “quargs”, but I like it.Vicki Boykis replied that she’s “… never said anything but “quargs”. This is like a parallel universe.”Some other amusing responses.This reminded me of a conversation I might have over beer at PyCon. or at a booth, or just standing around in the hallway.I miss so much the in person community. I’m grateful that little bits of it are intact on twitter.

    Joke:

    Pizza delivery fail

    Second joke:

    Why do you many developers use dark mode?Because bugs are attracted to light.
  • Sponsored by Techmeme Ride Home podcast: pythonbytes.fm/ride

    Special guest: Matthew Feickert

    Live streamed on YoutubeWatch on YouTube

    Brian #1: Introducing FARM Stack - FastAPI, React, and MongoDB

    Aaron BassetDescribes a todo CRUD application, available on github, to show the concepts in action.Animated gif showing how to use the FastAPI interactive documentation to understand the API in a browser.ShowsHow app connects to routes and endpoints.The run call with uvicorn to get an async even loop goingConnecting to a MongoDB databaseDefining models and how easy it is to set up a schema.Discusses routes and endpoints and how to hook up all the CRUD endpoints.The React bit is an application that changes as you modify the elements through the endpoint interactive session.Very cool way to work and get something running fast.

    Michael #2: py-applescript

    How would you like to automate your mac with Python rather than the dreadful AppleScript?py-applescript is an easy-to-use Python wrapper for NSAppleScript, allowing Python scripts to communicate with AppleScripts and AppleScriptable applications.Features:Scripts may be compiled from source or loaded from diskStandard 'run' handler and user-defined handlers can be invoked with or without argumentsArgument and result values are automatically converted between common Python types and their AppleScript equivalentsCompiled scripts are persistent: handlers can be called multiple times and top-level properties retain their stateAvoids any dependency on the legacy appscript library, flawed Scripting Bridge framework, or limited osascript executableThe applescript package exports four classes - AppleScript, ScriptError, AEType and AEEnum - plus one constant, kMissingValue.

    Matthew #3: airspeed velocity

    Are you developing a tool where performance is of individual functions is key? Do you want to be able to benchmark those metrics over every commit in your project with beautiful interactive visualizations?From the docs:> airspeed velocity (asv) is a tool for benchmarking Python packages over their lifetime. Runtime, memory consumption and even custom-computed values may be tracked. The results are displayed in an interactive web frontend that requires only a basic static webserver to hostDeveloped by a community of people that you’ll probably recognize if you hang around the SciPy and Jupyter ecosystem lead by Michael Droettboom and Pauli VirtanenThe docs include examples for benchmarks of NumPy, SciPy, and Astropy (example: SciPy)Example: SciPy’s interpolate.Interpolate1d.time_interpolate testProduces plot of timing performance across multiple parameterizations of the testClick on any node in the plot and get taken directly to the commit on GitHub! 🤯 Developed openly on GitHubUp on PyPI as asvpython -m pip install asvVery cute and very Pythonic joke in the test dashboards: Report title for library X is “airspeed velocity of an unladen X” 😛

    Brian #4: Parsecs experience with Hypothesis testing

    Parsec blog, Parsec is a client side encrypted file sharing service .Discussion of their experience with Hypothesis to test a 4 year old, large, asynchronous Python project.In particular, an algorithm around RAID5 redundancy.The test needs to check that:blocks can be split into chunksblocks can be rebuilt from chunksblocks can be rebuilt with one missing storage chunkGoes on to describe how they use a stateful mode of hypothesis.It’s a rather complex description. But very welcome.I’ve been waiting to hear how people are using Hypothesis for real world problems.Their recommendations for when to use Hypothesis:If you are writing a kind of encoder/decoder. Hypothesis is simply the perfect solution for you ;-)If you can write a simple oracle (or even better: you already have one available given you are reimplementing a protocol !)If the work is of type "hard to compute, simple to check"

    Michael #5: Amazon Web Services now offers Mac instances based on Mac mini; M1 coming later

    Amazon Web Services (AWS) is now offering Mac instances to developers who want to speed up building and testing of Mac, iOS, iPadOS, watchOS and tvOS apps.You no longer need a Mac and macOS to build iOS apps.The instances are based on physical Mac minis, and you get exclusive access to a machine while you are using it.Right now, Amazon is using Intel-based Mac minis, but the company says that it will have M1 Mac minis available in the first half of next year. Check out the blog post and the video below for more details.Good as a backstop until we get the newer faster M1 chips if you/your company is holding off.Although there's no virtualization in play here, the mac1.metal instances can be spun up and down nearly as rapidly, thanks to the AWS Nitro hardware management—which is invisible, from the customer's perspective.To someone who spins up a mac1.metal instance, the instance is for all intents and purposes a perfectly vanilla, brand-new Intel Mac mini.But there is also macstadium at 1/10th the price.

    Matthew #6: visidata

    Are you doing some data analysis and get given a new CSV file? Your first instinct is to load it up into pandas, right? Wouldn’t it be nice to drop in and explore at the command line though a bit to get the lay of the land and even do some preliminary analysis?From the docs:> Data science without the drudgery!> > VisiData is an interactive multitool for tabular data. It combines the clarity of a spreadsheet, the efficiency of the terminal, and the power of Python, into a lightweight utility which can handle millions of rows with easeSupports large number of file formats: 42Everything from CSV and JSON through JIRA, MySQL, and PNGDeveloped openly on GitHub by Saul PwansonOn PyPI as visidatapython -m pip install visidataThough as this is a Python application you probably going to want this installed regardless of the virtual environment you’re working in. Distributions of visidata are available through the package managers you’d expect: apt, (also nix, Guix but not yum?), Homebrew, and Conda-forge so can easily get to work on most Unix-based machines.Tested it with our trusty beloved tool pipx and that works too (thanks again Chad Smith!)python -m pip install pipxpipx ensurepathpipx install visidataSome pretty impressive live demo videos on the websitevisidata has come in quite handy for me on a frequent basis as I deal with JSON serializations of statistical models with hundreds of parameters. Having a way to explore those JSON files without getting visually overwhelmed is super nice!

    Extras

    Brian:

    PyCon US 2021

    Michael:

    PSF Fund RaiserTickets for PyCascades 2021 are now available on pretix. Our ticket prices are $10 for students, $20 for individuals, and $50 for professional attendees.Live streaming on YouTube!

    Matthew:

    Advent of Code 2020 has started. Happy coding! ❄️ If time (don’t want to hog or go over):Follow up from Episode 205:awkward v1.0.0rc1 is on PyPIpip install awkward should get what you ant by the time this show goes live (no more need for awkard0)Kudos to Python Bytes for making full transcripts of all the shows available for view on pythonbytes.fm — this helps be more inclusive of the deaf Python community (including Matthew’s good friend and coauthor)

    Joke:

    How to fix any computer - via theoatmean.com

  • The live stream recording on YouTube.

    Special guest: Jay Miller

    Sponsored by us! Support our work through:

    Our courses at Talk Python TrainingTest & Code PodcastPatreon Supporters

    Brian #1: Analyzing Kickstarter Campaigns with Python Data Science Tools

    Article title: “Kickstarter Projects — Do They Succeed?”Aditya PatkarUsing a Kaggle dataset of 378,661-ish projects up to 2018.Looks at using pandas data frames to explore the data.Using .describe() data frame method to learn a lot.Uses matplotlib and seaborn to analyze the data further.Odd statement that I’m not sure is straight faced or a really dry joke: “The data from 1970 seems to be bad or insignificant data.”Examples of using heat maps, line graphs, bar charts, to look at different aspects.Some results:35.64% of projects are successful (meaning goal hit)tech asks for the most for goals, and has the highest average per backer.Comics has the lowest pledged amount per backer average.Nice that you can use the techniques to ask your own questions of the data.

    Michael #2: GPU Accelerated Python for Machine Learning on Cross-Vendor Graphics Cards

    Building machine learning algorithms using the Vulkan Kompute Python FrameworkWhen you hear “CUDA”, that means Nvidia 🙂Uses Vulkan Kompute frameworkA large number of high profile (and new) machine learning frameworks such as Google’s Tensorflow, Facebook’s Pytorch, Tencent’s NCNN, Alibaba’s MNN —between others — have been adopting Vulkan as their core cross-vendor GPU computing SDK.As you can imagine, the Vulkan SDK provides very low-level C / C++ access to GPUs, which allows for very specialized optimizations.The main disadvantage is the verbosity involved, requiring 500–2000+ lines of C++ code to only get the base boilerplate required to even start writing the application logic.The Kompute Python package is built on top of the Vulkan SDK through optimized C++ bindings, which exposes Vulkan’s core computing capabilities. Kompute is the Python GPGPU framework.The main article talks through a couple of numerical computation examples.

    Jay #3: Adafruit PyPortal - CircuitPython Powered Internet Display

    Gift for the tinkering pythonistaCircuitPyUse it to make plenty of cool thingsScreen/speaker/Light Sensor Built-In

    Brian #4: Introduction to Linear Algebra for Applied Machine Learning with Python

    Pablo CaceresIntended as a reference and not a comprehensive review.Still, I very much appreciate it.Includes links to both free and paid resources to thoroughly learn linear algebraCoverssets, ordered pairs, relations, functions, vectorsmatriceslinear and affine mappingsmatrix decompositionUses numpy, pandas, and altair for examplesQuick (but useful) explanations of concepts, along with how to represent and do it with numpyI’m really just getting into it, but I’m enjoying it and this is the right level of handholding I needed.

    Michael #5: How many notebook frameworks? Many, and now +1 with Deepnote

    Deepnote is a new kind of data science notebook. Jupyter-compatible with real-time collaboration and running in the cloud. Free for individuals, paid for teams and companiesReal time collaboration is a key featureBuilt in versioning comingCode review in the notebook coming“View” your variables as a whole environmentBetter — real — autocompleteDashboards coming too

    Jay #6: imagekit.io

    image cdnstarted using imagekit on my own website and noticed faster load timesallows for some responsive “fanciness”Add BlursSmart CroppingPython API or URL-Schema

    Extras

    Michael:

    The Apple M1 mac mini wait continues. :)Talk Python To Me, pro editionPSF Fundraiser for the month of December: https://pythonbytes.fm/psf2020

    Jay:

    Elastic Community YouTube ChannelJust posted my lightning talk on looking at open data from the government.Upcoming interview on one of our newest clients - Eland which is python client to create pandas-like dataframes with elasticsearch datastores.My Podcast The PIT Show weekly insights from me on my developer journey and interviews with amazing folks in the tech space.Elastic Blog - Just posted my first Elastic Blog post Elastic Contributor Program: How to create a video tutorial

    Joke:

    via twitter.com/Spirix3/status/1330611989891207168

    Q: why can't SQL and NoSQL Developers date one other?A: because they don't agree on relationships.
  • Sponsored by us! Support our work through:

    Our courses at Talk Python TrainingTest & Code PodcastPatreon Supporters

    Michael #1: Running Python on .NET 5

    by Anthony ShawTalked about pyjion way back when on episode 49 with Brett Cannon..NET 5 was released on November 10, 2020. It is the cross-platform and open-source replacement of the .NET Core project and the .NET project that ran exclusively on Windows since the late 90’s. See the conference about it if you want to go deeper.Performance: I just saw a SO post about someone complaining their Python was 31x slower than C#.The most common way around this performance barrier is to compile Python extensions from C or using something like Cython..NET 5 CLR comes bundled with a performant JIT compiler (codenamed RyuJIT) that will compile .NETs IL into native machine instructions on Intel x86, x86-64, and ARM CPU architectures.Pyjion is a project to replace the core execution loop of CPython by transpiling CPython bytecode to ECMA CIL and then using the .NET 5 CLR to compile that into machine code.It then executes the machine-code compiled JIT frames at runtime instead of using the native execution loop of CPython.A few releases of Python ago (CPython specifically, the most commonly used version of Python) in 3.7 a new API was added to be able to swap out “frame execution” with a replacement implementation. This is otherwise known as PEP 523.This extension uses the same standard library as Python 3.9.Will this be compatible with my existing Python code? What about C Extensions?The short answer is- if your existing Python code runs on CPython 3.9 – yes it will be compatible.Tested against the full CPython “test suite” on all platforms. In fact, it was the first JIT ever to pass the test suite.Is this faster? The short answer a little, but not by much (yet).see also: https://twitter.com/anthonypjshaw/status/1328457723608928256?s=20

    Brian #2: PEP 621 -- Storing project metadata in pyproject.toml

    Progress on standardizing what goes into pyproject.tomlAuthors Brett Cannon, Paul Ganssle, Pradyun Gedam, Sébastien Eustace (of poetry), Thomas Kluyver (of flit), Tzu-Ping ChungMotivators of this PEP are:Encourage users to specify core metadata statically for speed, ease of specification, unambiguity, and deterministic consumption by build back-endsProvide a tool-agnostic way of specifying metadata for ease of learning and transitioning between build back-endsAllow for more code sharing between build back-ends for the "boring parts" of a project's metadataDoesn’t change any existing core metadataDoesn’t attempt to standardize all possible metadataIncluded in table named [project]:nameversiondescriptionreadmerequires-pythonlicenseauthors/maintainerskeywordsclassifiersurlsentry pointsdependencies/optional-dependenciesdynamic There’s an example in the PEP that helps clear things upMany items have synonyms specified for flit/poetry/setuptools (presumably for backward compatibility)

    Michael #3: GitHub revamps copyright takedown policy after restoring YouTube-dl

    In October following a DMCA complaint from the Recording Industry Association of America (RIAA) it was taken down at GitHub. Citing a letter from the Electronic Frontier Foundation (the EFF), GitHub says it ultimately found that the RIAA’s complaint didn’t have any merit. The RIAA argued the tool ran afoul of section 1201 of the US copyright law by giving people the means to circumvent YouTube’s DRM. the EFF dissects the RIAA’s claims, highlighting where the organization had either misinterpreted the law or how the code of YouTube-dl works. “Importantly, YouTube-dl does not decrypt video streams that are encrypted with commercial DRM technologies, such as Widevine, that are used by subscription videos sites, such as Netflix,” the organization points out when it comes to the RIAA’s primary claim.GitHub is implementing new policies to avoid a repeat of a repeat situation moving forward. First, it says a team of both technical and legal experts will manually evaluate every single section 1201 claim.If the company’s technical and legal teams ultimately find any issues with a project, GitHub will give its owners the chance to address those problems before it takes down their work.GitHub is establishing a $1 million legal defense fund for developers.Sidebar: EFF has just launched How to Fix the Internet, a new podcast mini-series that examines potential solutions to six ills facing the modern digital landscape.

    Brian #4: Install & Configure MongoDB on the Raspberry Pi

    Mark SmithDefinitely a “wow, I didn’t know you could do that” article.Tutorial walks throughInstalling 64 bit Ubuntu Server on a Raspberry PiConfigure wifiInstall MongoDB on PiSet up a user account, to safely expose MongoDB on a home network.Now you’ve got a MongoDB server in your house. So cool

    Michael #5: Extra! extra! extra!, hear all about it!

    Follow up on my critique of things like SQL & CSS put next to Python and Java. Maybe best to grab the conversation from here.Guido joins Microsoft, why? People seem to see this as a positive for sure. But they checked him out!New code editor roaming the streets: Nova from Panic.Two thumbs up on Big Sur and now waiting on the Mac Mini M1.

    Brian #6: A Python driven AI Stylist Inspired by Social Media

    Dale MarkowitzA bunch of Google tools (cloud storage, firebase, cloud vision api, product search api)Some React for front endPython to batch scriptGeneral oversimplified process:photos from social media for inspirationphotos of everything in your closet, multiple of each itemuse AI suggest outfits from your closet that match inspiration photosOk. The process is really more of a promo for Google AI products, and not so much about Python, but it’s a cool “look what you can do with software” kinda thing.Also, many of the tools used by online retail, like “similar products” and such, are available to lots of people now, and that’s cool.

    Joke:

    Back to the [dev] future!

  • Sponsored by us! Support our work through:

    Our courses at Talk Python TrainingTest & Code PodcastPatreon Supporters

    Brian #1: pip-chill - Make requirements with only the packages you need

    Ricardo BánffyLike pip freeze but lists only the packages that are not dependencies of installed packages.Will be great for creating requirements.txt files that look like the ones you would write by hand.

    I wish it had an option to not list itself, but pip-chill | grep -v pip-chill works.

    What do I have installed?

    (foo) $ pip freeze appdirs==1.4.4 black==20.8b1 click==7.1.2 mypy-extensions==0.4.3 ...No really, what did I myself install? (foo) $ pip-chill black==20.8b1 pip-chill==1.0.0Without versions? (foo) $ pip-chill --no-version black pip-chillWhat did those things install as dependencies? (foo) $ pip-chill -v --no-version black pip-chill # appdirs # Installed as dependency for black # click # Installed as dependency for black ...

    Michael #2: Windows update broke NumPy

    Sent in by Daniel MulkeyA recent Windows update broke some behavior that I think OpenBLAS (used by NumPy) relied on. There's a Developer Community thread here. I am a NumPy developer. We have been trying to track down a strange issue where after updating to windows 10 2004, suddenly code that worked no longer works. Here is the NumPy issue and here is the corresponding issue in OpenBLAS. The problem can be summarized: when calling fmod, something is changed so that much later calling an OpenBLAS assembly routine fails. The only difference I can see in the registers that visual studio exposes is that after the call to fmod, register ST(0) is set to NAN.Steve Dower and other Microsoft people have commented. The fix is slated to take until January 2021 to be released, though there are workarounds for some scenarios.Matt P. posted a workaround:For all those at home following along and looking for a quick fix, NumPy has released a bugfix 1.19.3 to work around this issue. The bugfix broke something else on Linux, so we had to revert the fix in release 1.19.4, but you can still install the 1.19.3 viapip install numpy==1.19.3.Note this is only works around the way this bug crashes NumPy (technically, in OpenBLAS which is shipped with NumPy), and may not fix all your problems related to this bug, Microsoft’s help is needed to do that.

    Brian #3: Build Plugins with Pluggy

    kracekumarBlog post related to talks given at PyGotham and PyCon IndiaPluggy is the plugin library used by pytestArticle starts with a CLI application that has one output format.Need is for more formats, implemented as plugins.Quick look at pluggy architecture of host/caller/core system and plugin/hook.Also plugin manager, hook specs, and hook implementations.Walks through the changes to the application needed to support plugins.I’ve been waiting for an article on pluggy, and this is nice. But I admit I’m still a little lost. I guess I need to watch one of the presentations and try to build something with pluggy.

    Michael #4: LINQ in Python

    via Adam: I seem to recall that Michael had a C# background, so this might be of interest:Bringing LINQ-like expressions to Python with linqitExample: last_hot_pizza_slice = programmers.where(lambda e:e.experience > 15) .except_for(elon_musk) .of_type(Avi) .take(3) # [[HTML_REMOVED], [HTML_REMOVED], [HTML_REMOVED]] .select(lambda avi:avi.lunch) # [[HTML_REMOVED], [HTML_REMOVED], [HTML_REMOVED]] .where(lambda p:p.is_hot() and p.origin != 'Pizza Hut'). .last() # [HTML_REMOVED] .slices.last() # [HTML_REMOVED]Also interesting asq: https://github.com/sixty-north/asq

    Brian #5: Klio : a framework for processing audio files or any binary files, at large scale

    Recently open sourced by SpotifyAn article about itKlio is based on Apache Beam and allowsintegration with cloud processing enginesopen graph of job dependenciesbatch and streaming pipelinesgoals:large-file input/outputscalability, reproducibility, efficiencycloser collaboration between researchers and engineersuses PythonObviously useful for Spotify, but they are hoping it will help with other audio research and applications.

    Michael #6: Collapsing code cells in Jupyter Notebooks

    via Marco GorelliYou mentioned in that episode that you'd like to have a way of collapsing code cells in Jupyter Notebooks so you can export them as reports - incidentally, I wrote a little blog post about how to do that - in case it's useful/of interest to you, here it is! Basically get a static HTML file that is the static notebook output but can start with the code cells collapsed and can toggle their visibility.

    Extras

    Michael:

    New Apple Silicon macs?Bot tweets: twitter.com/MichelARenard/status/1324269474544029696

    Joke:

    By Richard Cairns

    Q: Why did the data scientist get in trouble with Animal Welfare?A: She was caught trying to import pandas.

    “10e engineeeeeeeeers are the future.” - detahq

  • Sponsored by us! Support our work through:

    Our courses at Talk Python TrainingTest & Code PodcastPatreon Supporters

    Michael #1: fastapi-chameleon (and fastapi-jinja)

    Chameleon via Michael, Jinja via Marc BrooksConvert a FastAPI API app to a proper web appThen just decorate the FastAPI view methods (works on sync and async methods): @router.post('/') @fastapi_chameleon.template('home/index.pt') async def home_post(request: Request): form = await request.form() vm = PersonViewModel(**form) return vm.dict() # {'first':'Michael', 'last':'Kennedy', ...}The view method should return a dict to be passed as variables/values to the template.If a fastapi.Response is returned, the template is skipped and the response along with status_code and other values is directly passed through. This is common for redirects and error responses not meant for this page template.

    Brian #2: Django REST API in a single file, without using DRF

    Adam JohnsonHe’s been on Test & Code a couple times, 128 & 135Not sure if you should do this, but it is possible.Example Django app that is a REST API that gives you information about characters from Rick & Morty. Specifically, just Rick and Morty./ - redirects to /characters//characters/ - returns a JSON list/characters - redirects to /characters//characters/1 - returns JSON info about Rick/characters/2 - same, but for MortyShows off how with Django off the shelf, can do redirects and JSON output.Shows data using dataclasses. Hardcoded here, but easy to see how you could get this data from a database or other part of your system.

    Michael #3: 2020 StackOverflow survey results

    Most Popular TechnologiesLanguages: JavaScript (68%), Python (44%), Java(40%)Web frameworks: Just broken, jQuery? Seriously!?!Databases: MySQL (56%), PostgreSQL (36%), Microsoft SQL Server (33%), MongoDB (26%)Platforms: Windows (46%), macOS (28%), Linux(27%)Most loved languages: Rust, TypeScript, PythonMost wanted languages: Python, JavaScript, GoMost dreaded language: VBA & ObjectiveCMost loved DBs: Redis (67%), PostgreSQL (64%), Elasticsearch (59%), MongoDB (56%)Most wanted DBs: MongoDB (19%), PostgreSQL (16%)Most dreaded DB: DB2

    Brian #4: A Visual Guide to Regular Expression

    Amit ChaudharyGentle introduction to regex by building up correct mental models using visual highlighting.Goes through different patterns:specific characterwhite space (any whitespace \s, tab \t, newline \n)single-digit number \dword characters \w : lowercase, uppercase, digits, underscorethis sometimes throws me, since w seems like it might somehow be related to whitespace. It’s not.dot . : anything except newlinepattern negations:\d is digits, \D is anything that is not a digit\s whitespace, \S not whitespace\w word characters, \W everything elsecharacter sets with square brackets [], and optionally dash - for rangeanchors^ beginning of line$ end of lineescaping patterns with \repetition with {}, *, +, ?Using Python re modulefindallmatch and match.groupsearch

    Michael #5: Taking credit

    by Tim NoletOh @awscloud I really do love you! But next time you fork my OS project https://github.com/checkly/headless-recorder and present it as your new service, give the maintainers a short "nice job, kids" or something. Not necessary as per the APLv2 license, but still, ya know?Amazon CloudWatch Synthetics launches Recorder to generate user flow scripts for canariesA Chrome browser extension, to help you create canaries more easily.

    Brian #6: Raspberry Pi 400

    “complete personal computer, built into a compact keyboard”by itself, or as a kit with mouse and power adapter and cables and such, for $1004 core, 64-bit processor, 4 GB RAM, wifi & LAN, can drive 2 displays, 4K video40-pin GPIO header, so you can still play with hardware and such.There’s an adafruit video with Limor Fried where she describes this as something as close as we get today to an Apple IIe from my youth.For me, IIe was at school, at home I had a TRS80 plugged into an old TV and using my sisters tape deck for disk storage.This seems great for education use, but also as a second computer in your house, or a kids computer. Comes with a Beginner’s Guide that includes getting started with Python

    Extras:

    Brian:

    vim-adventures.com - with a dash. Practice vim key bindings while playing an adventure game. Super cool.

    Michael:

    TIOBE Index for November 2020 via Tyler Pedersen

    Joke:

    You built it, you run it.

  • Sponsored by Techmeme Ride Home podcast: pythonbytes.fm/ride

    Special guest: Steve Dower - @zooba

    Brian #1: Making Enums (as always, arguably) more Pythonic

    “I hate enums” Harry PercivalHilarious look at why enums are frustrating in Python and a semi-reasonable workaround to make them usable.Problems with enums of strings:Can’t directly compare enum elements with the valuesHaving to use .value is dumb.Can’t do random choice of enum valuesCan’t convert directly to a list of valuesIf you use IntEnum instead of Enum and use integer values instead of strings, it kinda works better.Making your own StringEnum also is better, but still doesn’t allow comparison.Solution: class BRAIN(str, Enum): SMALL = 'small' MEDIUM = 'medium' GALAXY = 'galaxy' def __str__(self) -> str: return str.__str__(self)Derive from both str and Enum, and add a *__str(self)__* method.Fixes everything except random.choice().

    Michael #2: Python 3.10 will be up to 10% faster

    4.5 years in the making, from Yury Selivanovwork picked up by Pablo Galindo, Python core developer, Python 3.10/3.11 release managerLOAD_METHOD, CALL_METHOD, and LOAD_GLOBAL improved“Lot of conversations with Victor about his PEP 509, and he sent me a link to his amazing compilation of notes about CPython performance. One optimization that he pointed out to me was LOAD/CALL_METHOD opcodes, an idea first originated in PyPy.”There is a patch that implements this optimizationBased on: LOAD_ATTR stores in its cache a pointer to the type of the object it works with, its tp_version_tag, and a hint for PyDict_GetItemHint. When we have a cache hit, LOAD_ATTR becomes super fast, since it only needs to lookup key/value in type's dict by a known offset (the real code is a bit more complex, to handle all edge cases of descriptor protocol etc).

    Steve #3: Python 3.9 and no more Windows 7

    PEP 11 -- Removing support for little used platforms | Python.orgWindows 7 - Microsoft Lifecycle | Microsoft DocsDefault x64 download

    Brian #4: Writing Robust Bash Shell Scripts

    David PashleySome great tips that I learned, and I’ve been writing bash scripts for decades.set -u : exits your script if you use an uninitialized variableset -e : exit the script if any statement returns a non-true return value. Prevents errors from snowballing.Expect the unexpected, like missing files, missing directories, etc.Be prepared for spaces in filenames. if [ "$filename" = "foo" ];Using trap to handle interrupts, exits, terminal kills, to leave the system in a good state.Be careful of race conditionsBe atomic

    Michael #5: Ideas for 5x faster CPython

    Twitter post by Anthony Shaw calling attention to roadmap by Mark ShannonImplementation plan for speeding up CPython: The overall aim is to speed up CPython by a factor of (approximately) five. We aim to do this in four distinct stages, each stage increasing the speed of CPython by (approximately) 50%: 1.5**4 ≈ 5Each stage will be targeted at a separate release of CPython. Stage 1 -- Python 3.10: The key improvement for 3.10 will be an adaptive, specializing interpreter. The interpreter will adapt to types and values during execution, exploiting type stability in the program, without needing runtime code generation.Stage 2 -- Python 3.11: Improved performance for integers of less than one machine word. Faster calls and returns, through better handling of frames. Better object memory layout and reduced memory management overhead.Stage 3 -- Python 3.12 (requires runtime code generation): Simple "JIT" compiler for small regions.Stage 4 -- Python 3.13 (requires runtime code generation): Extend regions for compilation. Enhance compiler to generate superior machine code.Wild conversation over here.One excerpt, from Larry Hastings:Speaking as the Gilectomy guy: borrowed references are evil. The definition of the valid lifetime of a borrowed reference doesn't exist, because they are a hack (baked into the API!) that we mostly "get away with" just because of the GIL. If I still had wishes left on my monkey's paw I'd wish them away (1).(1) Unfortunately, I used my last wish back in February, wishing I could spend more time at home.*

    Steve #6: CPython core developer sprints

    Hosted by pythondiscord.comhttps://youtu.be/gXMdfBTcOfQ - Core dev Q&A

    Extras

    Brian:

    Tools I found recently that are kinda awesome in their own way - Brianmcbroken.com - Is the ice cream machine near you working? just a funny single purpose websitevim-adventures.com - with a dash. Practice vim key bindings while playing an adventure game. Super cool.

    Joke:

    Hackobertfest 2020 t-shirthttps://twitter.com/McCroden/status/1319646107790704640

    5 Most Difficult Programming Languages in the World(Not really long enough for a full topic, but funny. I think I’ll cut short the last code example after we record)suggested by Troy CaudillAuthor: Lokajit Tikayatraymalboge, intercal, brainf*, cow, and whitespacewhitespace is my favorite: “Entire language depends on space, tab, and linefeed for writing any program. The Whitespace interpreter ignores Non-Whitespace characters and considers them as code comments.”Intercal is kinda great in that One thing I love about this article is that the author actually writes a “Hello World!” for each language.Examples of “Hello World!”malboge (=<`#9]~6ZY32Vx/4Rs+0No-&Jk)"Fh}|Bcy?`=*z]Kw%oG4UUS0/@-ejc(:'8dcintercal DO ,1 <- #13 PLEASE DO ,1 SUB #1 <- #238 DO ,1 SUB #2 <- #108 DO ,1 SUB #3 <- #112 DO ,1 SUB #4 <- #0 DO ,1 SUB #5 <- #64 DO ,1 SUB #6 <- #194 DO ,1 SUB #7 <- #48 PLEASE DO ,1 SUB #8 <- #22 DO ,1 SUB #9 <- #248 DO ,1 SUB #10 <- #168 DO ,1 SUB #11 <- #24 DO ,1 SUB #12 <- #16 DO ,1 SUB #13 <- #162 PLEASE READ OUT ,1 PLEASE GIVE UPbrain**ck (censored) ++++++++++[>+++++++>++++++++++>+++<<<-]>++.>+.+++++++ ..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.cow MoO MoO MoO MoO MoO MoO MoO MoO MOO moO MoO MoO MoO MoO MoO moO MoO MoO MoO MoO moO MoO MoO MoO MoO moO MoO MoO MoO MoO MoO MoO MoO MoO MoO moO MoO MoO MoO MoO mOo mOo mOo mOo mOo MOo moo moO moO moO moO Moo moO MOO mOo MoO moO MOo moo mOo MOo MOo MOo Moo MoO MoO MoO MoO MoO MoO MoO Moo Moo MoO MoO MoO Moo MMM mOo mOo mOo MoO MoO MoO MoO Moo moO Moo MOO moO moO MOo mOo mOo MOo moo moO moO MoO MoO MoO MoO MoO MoO MoO MoO Moo MMM MMM Moo MoO MoO MoO Moo MMM MOo MOo MOo Moo MOo MOo MOo MOo MOo MOo MOo MOo Moo mOo MoO Moowhitespace S S S T S S T S S S L T L S S S S S T T S S T S T L T L S S S S S T T S T T S S L T L S S S S S T T S T T S S L T L S S S S S T T S T T T T L T L S S S S S T S T T S S L T L S S S S S T S S S S S L T L S S S S S T T T S T T T L T L S S S S S T T S T T T T L T L S S S S S T T T S S T S L T L S S S S S T T S T T S S L T L S S S S S T T S S T S S L T L S S S S S T S S S S T L T L S S L L LAPL: (~R∊R∘.×R)/R←1↓ιR
  • Sponsored by us! Support our work through:

    Our courses at Talk Python TrainingTest & Code PodcastPatreon Supporters

    Michael #1: Awkward arrays

    via Simon ThorAwkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.This makes it better than numpy at handling data where e.g. the rows in a 2D array have different lengths. It can even be used together with numba to jit-compile the code to make it even faster.Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they’re not.Recently rewritten in C++ for the 1.0 release and can even be used from C++ as well as Python.Careful on installation: pip install awkward1 ← Notice the 1.

    Brian #2: Ordered dict surprises

    Ned Batchelder“Since Python 3.6, regular dictionaries retain their insertion order: when you iterate over a dict, you get the items in the same order they were added to the dict. Before 3.6, dicts were unordered: the iteration order was seemingly random.”The surprises:You can’t get the first item, like d[0], since that’s just the value matching key 0, if key 0 exists. (I’m not actually surprised by this.)equality and order (this I am surprised by)Python 3.6+ dicts ignores order when testing for equality{"a": 1, "b": 2} == {"b": 2, "a": 1}OrderdDicts care about order when testing for equalityOrderedDict([("a", 1), ("b", 2)]) != OrderedDict([("b", 2), ("a", 1)])

    Michael #3: jupyter lab autocomplete and more

    via Anders KällmarExamples show Python code, but most features also work in R, bash, typescript, and many other languages.Hover: Hover over any piece of code; if an underline appears, you can press Ctrl to get a tooltip with function/class signature, module documentation or any other piece of information that the language server providesDiagnostics: Critical errors have red underline, warnings are orange, etc. Hover over the underlined code to see a more detailed messageJump to Definition: Use the context menu entries to jump to definitionsHighlight References: Place your cursor on a variable, function, etc and all the usages will be highlightedAutomatic Completion: Certain characters, for example '.' (dot) in Python, will automatically trigger completionAutomatic Signature Suggestions: Function signatures will automatically be displayedRename: Rename variables, functions and more, in both: notebooks and the file editor.

    Brian #4: Open Source Tools & Data for Music Source Separation

    An online “book” powered by Jupyter BookBy Ethan Manilow, Prem Seetharaman, and Justin SalamonA tutorial intended to guide people “through modern, open-source tooling and datasets for running, evaluating, researching, and deploying source separation approaches. We will pay special attention to musical source separation, though we will note where certain approaches are applicable to a wider array of source types.”Uses Python and interactive demos with visualizations.Section “basics of source separation” that includes a primer on digitizing audio signals, a look time frequency representations, what phase is, and some evaluations and measurements.Includes use of a library called nussldeep learning approachesdatasetstraining deep networksBrian’s comments:Very cool this is an open source bookEven if you don’t care about source separation, the primer on waveform digitization is amazing.The interactive features are great.

    Michael #5: Pass by Reference in Python: Background and Best Practices

    Does Python have pointers?Some languages handle function arguments as references to existing variables, which is known as pass by reference. Other languages handle them as independent values, an approach known as pass by value.Python uses pass by assignment, very similar to pass by ref.In languages that default to passing by value, you may find performance benefits from passing the variable by reference insteadIf you actually want to change the value, considerReturning multiple values with tuple unpackingA mutable data typeReturning optional “value” typesFor example, how would we recreate this in Python?public static bool TryParse (string s, out int result);Tuple unpacking def tryparse(string, base=10): try: return True, int(string, base=base) except ValueError: return False, None success, result = tryparse("123")Optional types: def tryparse(string, base=10) -> Optional[int]: try: return int(string, base=base) except ValueError: return None if (n := tryparse("123")) is not None: print(n)Best Practice: Return and Reassign

    Brian #6: Visualizing Git Concepts

    by onlywei Wei WangGit Basics is good, and important, but hard to get all these concepts to sink in well until you play with it.Visualizing Git Concepts with D3 solidifies the conceptsPractice using git commands without any code, just visualizing the changes to the repository (and sometimes the remote origin repository) while typing commands.commit, branch, checkout, checkout -breset, revertmerge, rebasetagfetch, pull, pushIncredibly powerful to be able to play around with these concepts without using any code or possibly mucking up your repo.

    Extras:

    Brian:

    micro:bit now has a speaker and a microphone - available in November

    Michael:

    Firefox containersTwitch!

    Joke:

    Q: Where do developers drink?A: The Foo bar

    - Knock Knock!- An async function- Who's there?

  • Sponsored by Techmeme Ride Home podcast: pythonbytes.fm/ride

    Special guests

    Carlton GibsonWilliam Vincent

    Brian #1: nbQA : Quality Assurance for Jupyter Notebooks

    Sent in by listener and Patreon supporter (woohoo!!!) Marco Gorelli.We’ve now talked about running black on Jupyter notebooks in the past (at least 2) shows?Marco’s recommendation is nbQAnbQA lets you run all this on notebooks:blackisortmypyflake8pylintpyupgrade to upgrade syntax doctest to check examples Run as a pre-commit hookConfigure in pyproject.tomlAlso (from Marco) better than standalone black due to:can run on a directory, not just one file at a timekeeps diffs minimal and easier to read then blackpreserves trailing semicolons, as they are used to suppress output in notebookssupports most standard magic commandsAnd the nbQA project is tested with …. drum roll …. pytest (of course)

    Michael #2: The PSF yearly survey is out, go take it now!

    This is the fourth iteration of the official Python Developers Survey.The results of this survey serve as a major source of knowledge about the current state of the Python communityTakes about 10 minutesThey will randomly choose 100 winners (from those who complete the survey in its entirety), who will each receive an amazing Python Surprise Gift Pack.Analysis is really well done, see the 2019 results.

    Will #3: From Prototype to Production in Django

    Django defaults to local/prototype settings initially when run startproject command. settings.py file contains global configs for a project. What needs to change for production?DEBUG set to FalseSECRET_KEY actually secret and not in source controlALLOWED_HOSTSDatabase probably not SQLite Configure static/media filesChange admin path away from /adminUser registration, typically use django-allauthEnvironment variables preferred method to have local/production settings. environs is Will’s favorite 3rd party package but multiple others.Use Django deployment checklist aka python check --``deploy, add HTTPS all over basicallyDJ Checkup websiteWhat else? Testing, logging, performance, security, etc etc etcDjango for Professionals book covers all of this and more including Docker

    Carlton #4: Deployment: Getting your app online

    Deployment (and how hard it it) seem to come up almost every week on Django Chat. I think a lot about The Deployment Gap that exists: a new user finishes the Django tutorial, or the DRF tutorial or Will’s Django for Beginners book, and they’re still a long way from being able to deploy. PaaS look like a good option. (Heroku/App Service/DO’s new one, and so on) but they’re a bit of a cul-de-sac (do you use that in America?) — you drive in, get to the end and then have to drive out again.On the other hand, DIY all looks far-too-scary™: there’s provisioning servers, private clouds, firewalls, permissions, block stores, objects stores, … — Argh! (On top of all the usual DNS, and all the rest of it.) Plus there’s a tendency I think towards fashion: you’d think you can’t possibly deploy without adopting a micro-service architecture, or a container orchestration platform. It’s too much.This is same as, You couldn’t possibly use Django, Postgres… — you have to use the New Hotness™. I think there’s a simpler story: start with a VM, a relational database, a simple network setup, and grow from there. There’s still moving parts, but it’s not that complex.I’m working on a tool for all this, Button. It’s coming in 2021. It’s a simpler deployment story. It’s part tool, part guide. It get’s you through the Argh! It’s too scary bit. You can sign up for early updates at https://btn.dev

    Brian #5: All Contributors“This is a specification for recognizing contributors to an open source project in a way that rewards each and every contribution, not just code.The basic idea is this:

    Use the project README (or another prominent public documentation page in the project) to recognize the contributions of members of the project community.

    People are giving themselves and their free time to contribute to open source projects in so many ways, so we believe everyone should be praised for their contributions (code or not).”

    used by nbQAIt is a specification for how to be consistent in listing contributors.Also includes an Emoji Key, to be used with contributors name (and optionally avatar) to denote the kind of contribution they’ve made:💻 code📖 doc🎨 design💡 example🚧 maintenance🔌 pluginand many, many moreAnd a GitHub bot to automate acknowledging contributors to your open source projects.Uses natural language parsing to add people as contributors and add the appropriate emojiAlso includes a cli for adding contributors, comparing GitHub contributors to your listed contributors, and more.

    Michael #6: MovingPandas

    A Python library for handling movement data based on Pandas and GeoPandasIt provides trajectory data structures and functions for analysis and visualization.MovingPandas development started as a QGIS plugin idea in 2018. But made more sense as its own libraryFeaturesConvert GeoPandas GeoDataFrames of time-stamped points into MovingPandas Trajectories and TrajectoryCollectionsAdd trajectory properties, such as movement speed and directionSplit continuous observations into individual tripsGeneralize TrajectoriesAggregate TrajectoryCollections into flow mapsCreate static and interactive visualizations for data explorationMovingPandas makes it straightforward to compute movement characteristics, such as trajectory length and duration, as well as movement speed and direction.Example df = pd.DataFrame([ {'geometry':Point(0,0), 't':datetime(2018,1,1,12,0,0)}, {'geometry':Point(6,0), 't':datetime(2018,1,1,12,6,0)}, {'geometry':Point(6,6), 't':datetime(2018,1,1,12,10,0)}, {'geometry':Point(9,9), 't':datetime(2018,1,1,12,15,0)} ]).set_index('t') gdf = gpd.GeoDataFrame(df, crs=CRS(31256)) traj = mpd.Trajectory(gdf, 1)

    Extras

    Carlton:

    Listen to Django ChatCheck out Will’s Tutorials and BooksSign up for early updates on Button

    Michael:

    Transcripts are back!Trying to live in DuckDuckGo land. Crazy or smart? :)And remember to do your periodic Google TakeoutDeployed my first Fast API web API & site

    Joke:

    “Give a person a program, frustrate them for a day. Teach them to program, frustrate them for a lifetime. 🙂” (…unless you teach them to test at the same time. - Brian)

    The failed interview: “Sorry, we’re looking for someone aged 22-26… with 30 years of experience with Flask”

  • Sponsored by DataDog: pythonbytes.fm/datadog

    Michael #1: Introducing DigitalOcean App Platform

    Reimagining PaaS to make it simpler for you to build, deploy, and scale apps.Many of our customers have come to DigitalOcean after their PaaS became too expensive, or after hitting various limitations.You can build, deploy, and scale apps and static sites by simply pointing to your GitHub repository. Built on DigitalOcean Kubernetes, the App Platform brings the power, scale, and flexibility of Kubernetes without exposing you to any of its complexity.App Platform is built on open standards providing more visibility into the underlying infrastructure than in a typical closed PaaS environment.You can also enable ‘Autodeploy on Push,’ which automatically re-deploys the app each time you push to the branch containing the source code.To efficiently handle traffic spikes (planned or unplanned), the App Platform lets you scale apps horizontally (i.e., add more instances that serve your app) and vertically (beef up the instances with more CPU and memory resources). (with zero downtime)What can you build with the App Platform? Web apps, Static sites, APIs, Background workers

    Brian #2: Announcing Playwright for Python

    playwright-pythonplaywrignt-pytestit’s a Microsoft thingthe pitch: “With the Playwright API, you can author end-to-end tests that run on all modern web browsers. Playwright delivers automation that is faster, more reliable and more capable than existing testing tools.”timeout-free automationautomatically waits for the UI to be readyIntended to stay modernemulation of mobile viewportsgeolocationweb permissionscan automate scenarios across multiple pagescross browserChromium (Chrome and Edge), WebKit (Safari), and FirefoxSafari rendering even works on Windows and Linuxpytest compatibleDjango compatibleCan work within CI/CD, even GH actions.

    Michael #3: Asynchronously Opening and Closing Files in asyncio

    Article by Chris Wellonsasyncio has support for asynchronous networking, subprocesses, and interprocess communication. However, it has nothing for asynchronous file operations — opening, reading, writing, or closing.If a file operation takes a long time, perhaps because the file is on a network mount, then the entire Python process will hang.Let’s build it!The usual way to work around the lack of operating system support for a particular asynchronous operation is to dedicate threads to waiting on those operations. By using a thread pool, we can even avoid the overhead of spawning threads when we need them. Plus asyncio is designed to play nicely with thread pools anyway.open() uses with so build an aopen() to have async with. Here’s the tasty bit: def __aenter__(self): def thread_open(): return open(*self._args, **self._kwargs) loop = asyncio.get_event_loop() self._future = loop.run_in_executor(None, thread_open) return self._futureaiofile package

    Brian #4: Excel: Why using Microsoft's tool caused Covid-19 results to be lost

    this article was on bbc.com, but it was in several placesNearly 16,000 coronavirus cases went unreported in England.Logs pulled together from data from commercial testing firms (filed as csv files) was combined in a Excel xls template so that it could then be uploaded to a central system and made available to the NHS Test and Trace team, as well as other government computer dashboards.XLS was one problem. Limit is about 65k rows.XLSX increases that limit by about 16 times.But still, …. Excel for this?Comment from Prof Jon Crowcroft from the University of Cambridge:"Excel was always meant for people mucking around with a bunch of data for their small company to see what it looked like.”“And then when you need to do something more serious, you build something bespoke that works - there's dozens of other things you could do.”"But you wouldn't use XLS. Nobody would start with that."In short: Best practices in computing don’t always make it into the rest of the world. Much of the world still runs on Excel.What does this have to do with Python? Well.. Big datasets should use databases and Python. Check out the Talk Python free webcast on moving from Excel to Python: talkpython.fm/excel-webcast

    Michael #5: locust.io

    via Prayson Daniellocust.io is awesome tool to simulate users hammering your endpoint. Quite handy.An open source load testing tool: Define user behavior with Python code, and swarm your system with millions of simultaneous users. Usage: after installing it via pip, you can map your local endpoint locust --host=http://localhost:5000 and open http://localhost:8089 to access the locust web ui to simulate usageFeatures:Define user behavior in code: No need for clunky UIs or bloated XML. Just plain code.Distributed & scalable: Locust supports running load tests distributed over multiple machines, and can therefore be used to simulate millions of simultaneous usersProven & battle tested: Locust has been used to simulate millions of simultaneous users. Battlelog, the web app for the Battlefield games, is load tested using Locust, so one can really say Locust is Battletested ;).Example: from locust import HttpUser, between, task class WebsiteUser(HttpUser): wait_time = between(5, 15) def on_start(self): self.client.post("/login", { "username": "test_user", "password": "" }) @task def index(self): self.client.get("/") self.client.get("/static/assets.js") @task def about(self): self.client.get("/about/")

    Brian #6: Fixing Hacktoberfest

    various sourcesHacktoberfest is an interesting idea sponsored by Digital Ocean, and other sponsors.Overall, it’s a good idea. Encourage people to contribute by bribing them with a t-shirt and other swag.Problem and some solutions outlined well by Anthony Sottile in what’s (wrong with) hacktoberfest?There’s always been some spam associated with hacktoberfest. Tiny bizarre PRs, PRs to unmaintained repos, etc.This year has been worseA fairly popular YouTuber posted a video showing people how to get a free t-shirt by doing things like adding “- an awesome project” or expanding “It’s” to “It is” to the readme, then submitting it as “improved docs”.Changes:On 10/3, rules changed: An update on efforts to reduce spam with Hacktoberfest: introducing maintainer opt-in and moremaintainers can opt-in by adding hacktoberfest topic to their repo.No longer have to opt outShould discourage spamming of inactive reposSummary: PRs count if: > Submitted during the month of October AND ( > The PR is labelled as hacktoberfest-accepted by a maintainer OR > Submitted in a repo with the hacktoberfest topic AND ( > The PR is merged OR > The PR has been approved > ) > )- The deadline for completions, merging, labeling, and approving is November 1.- I applaud DO and whoever else is working on hacktoberfest for reacting quickly to this.

    Extras:

    Michael:

    PyCascades 2021 will take place on Saturday, February 20th from many locations across the Pacific Northwest and beyond. Call for Proposals 📣 PyCascades has been lucky to give our stage to incredible speakers with wonderful talks over the last three years. We are really looking forward to showcasing our community again next year. Our Call for Proposals (CFP) opens today and closes at the end of the day on Tuesday, November 10th, 2020 Anywhere on Earth.Patricio Reyes, a researcher at Barcelona Supercomputing Center (virtual tour): You could also consider talking about nb_black: a simple black formatter for Jupyter and JupyterLab too.There is another project (only for JupyterLab): JupyterLab Code Formatter: jupyterlab-code-formatter.readthedocs.io

    Joke: More Classical Programmer Paintings

    “Delivering a feature in the time of a codefreeze”“RHEL sysadmins entering the Docker convention floor”