tag:protips.maxmasnick.com,2013:/posts Dev pro tips 2017-06-12T15:18:34Z by Max Masnick tag:protips.maxmasnick.com,2013:Post/1163180 2017-06-12T15:18:20Z 2017-06-12T15:18:34Z Fix broken camera on macOS without restarting

Sometimes the built-in camera on my Mac will stop working (no green light; no video). Restarting will fix it, but this appears to also work: sudo killall VDCAssistant

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1158719 2017-05-30T03:15:48Z 2017-05-30T03:15:48Z Disabling the startup chime on macOS Sierra

From How-To Geek:

Run sudo nvram SystemAudioVolume=%80 in the Terminal to disable the startup chime. Works like a charm on my late 2014 iMac.

To re-enable, run sudo nvram -d SystemAudioVolume (I didn't test this part).

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1153468 2017-05-12T11:12:34Z 2017-05-12T11:12:34Z macOS: Set audio balance to middle to correct drift with an AppleScript

On multiple Macs, I have had the left/right balance of my headphones randomly drift. Apple says this can be caused by changing the volume while the CPU is under load, which doesn't seem to be what is happening to me. In any case, it's annoying.

Here's an AppleScript that will automatically reset the sound balance to the middle for the current output device:

There's also a $5 app called Balance Lock that in theory will solve this problem – I haven't tested it.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1152686 2017-05-09T11:34:46Z 2017-05-09T11:34:46Z Loading multiple MySQL result sets into Pandas DataFrames

Note: This could be done in a loop. I don't do this because I often have results set-specific code that I want to run (e.g. renaming variables).

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1150522 2017-04-30T20:21:17Z 2017-04-30T20:21:17Z Darkening gray text with ImageMagick

The following ImageMagick command will take an image with gray text (and a white background) and make the text black.

magick convert -level 80%,100%,1 in.png out.png

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1149147 2017-04-25T12:11:34Z 2017-04-25T12:11:34Z 2x2 tables in Jupyter/IPython notebooks

I've written some helper methods for quickly generating 2x2 tables (or contingency tables, cross tabulations, crosstabs) in Jupyter/IPython notebooks. This achieves the same thing, more or less, as Stata's tabulate twoway command.

The notebook below has the helper methods and examples of their output. I've added these to my ipython-setup file because I use them so frequently.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1103156 2016-10-28T18:23:11Z 2016-10-28T18:23:11Z Hide all gifs with CSS
img[src*="gif"] {
    display: none;
}
]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1085541 2016-09-01T04:20:02Z 2016-09-01T04:20:02Z Get SHA1 hash for every file in a folder on OS X

cd path/to/folder
ls * | xargs openssl sha1

(Source)

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1081438 2016-08-15T20:01:18Z 2016-11-15T11:51:42Z Python: Get git hash for the last commit modifying the current folder
from os import path
p = path.dirname(path.abspath(__file__))
hash = subprocess.check_output(["git", "rev-list", "-1", "HEAD", "./"], cwd=p).strip()
]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1078612 2016-08-03T22:39:01Z 2017-02-10T03:33:52Z Querying MySQL from Stata

Step 0: Install the ODBC driver

  1. Download and install the iODBC driver manager. The next step will fail if this isn't installed first.
  2. Download and install the ODBC driver.

Step 1: Set up ODBC driver

See these instructions. My ODBC configuration (~/.odbc.ini) looks like:

[ODBC Data Sources]
mysql = MySQL ODBC 5.3 ANSI Driver

[ODBC]
Trace         = 0
TraceAutoStop = 0
TraceFile     = 
TraceLibrary  = 

[mysql]
Driver      = /usr/local/lib/libmyodbc5a.so
Description = desc here
SERVER      = localhost
PORT        = 3306
USER        = root
PASSWORD    = passwordhere
DATABASE    = databasehere

Make sure your driver location is correct. It could also be something like /usr/local/mysql-connector-odbc-5.3.7-macos10.12-x86-64bit/lib/libmyodbc5a.so.

Step 2: Make a ODBC call from Stata

For example:

clear
set odbcdriver ansi
odbc list
// Option 1
odbc load, table("tablename") dsn("mysql") clear noquote
// Option 2
odbc load, exec("select * from tablename") dsn("mysql") clear noquote

Using a plugin is another option, but I haven't tried this.

Update: Loading in SQL from an external file


This allows you to do something like:

cd "/path/to/your/folder"
clear
set odbcdriver ansi
odbc list
loadsql using your-sql.sql, dsn(mysql)

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1063375 2016-06-14T17:01:41Z 2017-04-03T22:25:24Z Literate Python setup with pweave and Atom

In the past I’ve used IPython/Jupyter notebooks for literate programming, but writing code in the browser is not a great experience and I was having terrible environment issues. I started looking at alternatives, and settled on trying out pweave. It’s the same idea as IPython, allowing me to intersperse code and output with prose, but rather than using the browser I can use a standard programming editor. (It also solves my environment problem.)

The setup for getting a pweave workflow that still has all the nice parts of an IPython notebook is a little involved, so here’s what I did:

  • Get the Atom editor (I generally use Sublime Text, but it can’t run Python inline)

  • Set up a virtualenv and pip install "ipython[notebook]" Pweave

  • Install the following Atom packages:
  • Make the following configuration changes:
    • Append to styles.less:

      // Hydrogen output - font size is too small  
      .hydrogen.output-bubble pre {  
        font-size: 16px !important;  
      }
      // Hydrogen - hack for 2x images  
      .hydrogen.output-bubble .bubble-result-container img {  
        width: 50% !important  
      }
      
    • Append to config.cson:

      ".md.pweave.source":  
        "expand-region":  
          commands: [  
            {
              command: "expand-region:select-inside-back-ticks"  
              recursive: true  
            }
          ]
      
    • Append to keymap.cson:

      '.platform-darwin atom-text-editor':  
        'cmd-enter': 'hydrogen:run',  
        'cmd-shift-space': 'expand-region:expand'
      
    • In the settings under Settings > Packages > Hydrogen:
      • Kernel Mappings (this enables support for .pmd files): {"pweave markdown": "Python 2"}

      • Startup Code:

        {"Python 2": "import matplotlib as matplotlib_import_only\nmatplotlib_import_only.use('Agg')\n%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\npython=None"}
        

        This does a few hacky things:

        • import matplotlib as matplotlib_im ... rt_only.use('Agg'): fix for this issue with a bouncing rocketship dock icon
        • %matplotlib inline: display figures inside Atom rather than in a separate window
        • %config InlineBackend.figure_format = 'retina': turns on high-resolution figures, which are displayed by Atom as double-sized low-resolution figures without the hack in styles.less above
        • python=None: a hack to avoid an error when automatically selecting and running a block of Python code

Some of this configuration is necessary to just get hydrogen to run Python code.

The rest fixes a major problem with pweave: it’s easy to run one line of code, and it’s easy to run all the code in a file, but how do I run only one block of code? (This is equivalent to being able to running an entire cell in IPython with shift-enter.)

The solution is to use the expand-region plugin to select the current block of code, and the python=None hack to avoid an error when it selects the “python” in ```python.

Here’s what it looks like in action:


To convert the .pmd file to a .html that can be shared, simply run pweave -f md2html test.pmd. Here's what the output from the above example looks like.


Notes

  • Remember to open Atom from the command line after activating your virtualenv (see the "Usage" section here).
  • In order to get the pweave command to run with my virtualenv, I manually edited the shebang in /usr/local/bin/pweave to directly point to the Python binary in my virtualenv: #!/Users/max/.virtualenvs/data/bin/python
]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1050816 2016-05-13T15:22:48Z 2016-05-13T15:22:48Z Reserved words lists in Ruby/Rails, Python, MySQL, and PostgreSQL ]]> Max Masnick tag:protips.maxmasnick.com,2013:Post/1048946 2016-05-09T17:41:00Z 2016-05-09T17:41:00Z Disable "mirror screen" keyboard shortcut (Command+F1 or Command+Brightness down)

Thanks to this Ask Different post and Karabiner, I've been able to disable the very annoying "mirror screen" keyboard shortcut.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1045660 2016-05-02T18:12:40Z 2016-05-26T18:03:29Z Keyboard Maestro: Convert Markdown to JIRA Markup

JIRA uses its own plain-text markup syntax, which I don't like as much as Markdown.

Fokke Zandbergen has a great online service for converting between JIRA markup and Markdown, but it's too cumbersome to use this when you are living in JIRA comments.

I re-appropriated Fokke's JavaScript into a Keyboard Maestro macro:

This will take the Markdown in the clipboard and paste it as JIRA markup whenever you type ";jira".

You can download the macro here.

Update – May 26, 2016: I have added support for multi-level unordered lists – the updated code is below.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1038871 2016-04-22T15:08:19Z 2016-04-22T15:08:48Z Move newest file in Downloads folder to the frontmost Finder window

Here's what you do:

  1. Open the folder where you want to move your newest download in Finder.
  2. Trigger the shell script below.

The file will magically be moved! You'll even get a Notification Center notification.

The shell script

Triggering the shell script

I use an AppleScript saved as a .app and triggered with LaunchBar. Here's the AppleScript:

do shell script "bash /Users/YOUR_USERNAME/path/to/script.sh"

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1038866 2016-04-22T15:00:45Z 2016-04-22T15:00:45Z Switching from TextExpander to Keyboard Maestro

The makers of TextExpander recently changed their pricing model with the release of v6 from paying once for each version to a subscription. I have no problem paying for software that makes me more productive, as TextExpander certainly does.

However, v6 of TextExpander started inserting non-breaking spaces rather than normal spaces at the beginning of Python code snippets: I would paste code into TextExpander with 4 space characters in a row, and it would come out with 3 NBSPs and one normal space. This effectively broke TextExpander for some of my most-used snippets.

Additionally, I've always had quality and stability issues with TextExpander and Smile's other product, PDFpenPro. For example, in TextExpander v5, it was impossible to tab from the snippet entry field to the abbreviation field. This infuriating problem was left unresolved for all of v5, but was thankfully fixed in v6. And PDFpenPro is simply much less stable than the alternatives (Preview.app and PDF Expert), with fairly frequent beachballs and freezes.

TextExpander v6 requires syncing to their central server. Given the software quality issues with their stand-alone apps, I'm very concerned about snippet loss from sync errors. I'm also not convinced that the QA issues with their native applications will ever be resolved, especially with the added burden of their new sync/subscription business model. And I am skeptical that their new business model will be sustainable long-term, given these other issues and the response of many users to their subscription pricing.

I decided it was time to look for an alternative.

Keyboard Maestro

Keyboard Maestro is general-purpose Mac automation software, which also has the capability to expand text snippets like TextExpander. I manually migrated my snippets over in about an hour (it can be done automatically too).

What I like

  • Keyboard Maestro (KM) seems generally more stable and performant than TextExpander (TE).
  • KM's two column UI for listing snippets is much more effective than TE's single column with collapsable headings.
  • KM allows multiple snippets with the same shortcut: when you type a conflicted shortcut, it shows a pop-up menu where you can select the snippet you want.
  • KM does lots of things besides snippets, like letting you ctrl-tab between multiple windows in the same application!

What I don't like

  • The KM quick search doesn't show the shortcut for the snippet (TE's quick search does).
  • Setting up new snippets in KM is a bit more complicated than with TE. This is because KM is a multi-purpose tool, and there are definitely benefits to specialization. But at least the tab key works as expected.

Other than these minor complaints, I've had zero problems with KM.

Setting up text expansion in Keyboard Maestro

Here's what my text expansion snippets ("macros" in KM parlance) look like – it seems like it's necessary to reset the system clipboard after expanding the snippet if you use the "paste" insert method rather than the "type" method:

Implications for this website

I've posted a number of tips on this site that use TextExpander. So far I have been able to convert these to Keyboard Maestro macros with minimal effort.

From now on, I will be posting KM macros rather than TE snippets.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1023170 2016-04-01T01:18:44Z 2016-04-01T22:43:12Z TextExpander Snippet: Path to Selected Finder Item

Thanks to this post on the Keyboard Maestro forum:

Update:

Right click on any Finder item, then hold Option. “Copy” becomes “Copy as Pathname”!!

(Thanks @anatomyofashane - you are a genius.)
@masnick

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1020549 2016-03-27T19:14:21Z 2016-03-27T19:14:21Z Stata: "clear" results

From Dan Menes on the Stata listserv:

display _newline(200)

It doesn't technically clear the results window, but it does allow you to easily add a visual break in the results. I recommend putting this in a TextExpander snippet for easy access.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/1018131 2016-03-22T20:26:56Z 2016-03-22T20:26:56Z TextExpander: Snippet to Get the URL of the Active Chrome Tab

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/998874 2016-02-22T14:01:14Z 2016-02-22T14:01:40Z Bookmarklet for Removing GIFs from a Web Page

It replaces all .gif files with a photo of a kitten. It requires jQuery (for now).

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/974411 2016-01-20T02:39:17Z 2016-01-20T02:39:17Z Stata: Compress All .dta Files in a Folder

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/915888 2015-10-12T17:07:41Z 2015-10-12T17:07:41Z Download Files (Like PDFs) Open in Safari to the Current Finder Folder

I often click PDF links in Safari, and the PDF will open in that Safari window rather than downloading. This is ok with me. I don't like a bunch of files cluttering my ~/Downloads folder.

But when I do want to download one of these PDFs, there isn't an easy way to do it. There is no "Save As..." in the right click menu, and the HUD buttons at the bottom of the viewer are hard to reliably bring up.

Enter this Automator workflow, which I trigger with LaunchBar. It will download the file (PDF or otherwise) in the frontmost Safari window into the frontmost Finder window's folder, or to the Desktop if no Finder windows are open. Download it here.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/908786 2015-09-23T18:29:12Z 2015-09-23T18:29:12Z List of Ruby's % operators

Compiled from this Stack Overflow answer, here is the list of Ruby’s very useful % operators:

  • %w() creates an array from space-delimited strings (a b c becomes [‘a’, ‘b’, ‘c’])
  • %r() is another way to write a regular expression
  • %q() is for single-quoted strings (so you can do %q(that's right) without escaping the '
  • %Q() is for double-quoted strings so you don't have to escape ", but can do interpolation (%Q(#{1+1}) => "2")
  • %x() is a shell command
  • %i() gives an array of symbols (Ruby >= 2.0.0)
  • %s() turns foo into a symbol (:foo)
]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/904777 2015-09-13T21:50:14Z 2015-09-13T21:50:14Z How to Strip Photo Metadata in the Terminal with exiftool

If you want to strip metadata from photos (e.g., location data from your phone's GPS), there are a few ways to do it:

  • On iOS, I recommend Metapho.
  • On Mac, you can use the built-in Preview.app, and there are lots of apps in the App Store of dubious quality that do this.

There's another option for Mac if you want to use the Terminal: exiftool

You can get exiftool with brew install exiftool, and then you can use this command to strip all the metadata from a photo:

exiftool -all= /path/to/file.jpg

You can verify that it worked with exiftool -a /path/to/file.jpg | grep GPS. Try running this before and after the previous command, and you can see how the GPS info is removed.

I have added the following to my .zshrc file for quick access to these commands:

alias exifstrip='exiftool -all='

function exifgps(){
    exiftool -a "$1" | grep GPS
}

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/891596 2015-08-08T19:11:49Z 2015-08-08T19:11:49Z Bookmarklet to Search Amazon.com Order History

Go here to get the bookmarklet.

Here's the code:

var x = window.prompt("Search Amazon.com order history for:");
window.location.href='https://www.amazon.com/gp/your-account/order-history/ref=oh_aui_search?opt=ab&search='+encodeURIComponent(x);

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/887889 2015-07-29T17:03:34Z 2015-07-29T17:03:34Z IPython Notebook Setup Code

I just posted the code I use at the top of every IPython notebook to GitHub: https://github.com/masnick/ipython-setup

It was somewhat difficult to figure out what to import (and the conventions for this) when starting with IPython. This setup file takes care of that.

It also has some additional goodies like my code toggle button and my method to display raw HTML inline in the notebook.

In any case, I recommend standardizing the contents of the first cell in IPython notebooks, saving this out to a file, and then using %load magic to bootstrap the first cell. (This is explained in more detail in the GitHub link above.)

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/887875 2015-07-29T16:02:29Z 2015-07-29T16:06:55Z SciPy: Using GridSpec to Display Box Plots Under Histograms
Here's how to get output like this using matplotlib and its GridSpec class:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gs

%matplotlib inline

df = pd.DataFrame([
        np.random.randn(1000),
        np.random.random_integers(1, 2, 1000)
    ]).T
df.columns = ["value", "group"]


fig = plt.figure(figsize=(10, 6))
grid = gs.GridSpec(2, 2, height_ratios=[3, 1]) 
for i, g in enumerate([1, 2]):
    
    subset = df['value'][df['group'] == g]
    
    # Histogram
    ax = plt.subplot(grid[i])
    ax.hist(list(subset), color='k', alpha=0.4)
    ax.set_title("Group %s" % g)
    
    # Box plot
    ax2 = plt.subplot(grid[i+2])
    pd.DataFrame(subset).boxplot(vert=False, return_type='axes')
    ax2.set_yticklabels([''])

fig.tight_layout()
fig.subplots_adjust(top=0.85)
fig.suptitle('Example', fontsize=20)

None # Don't display the last thing -- `%matplotlib inline` will display the graphs no matter what.


It looks like Gist embed is broken here on PostHaven for IPython notebooks, so the embed below may look strange:

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/885198 2015-07-22T22:52:01Z 2015-07-22T22:52:01Z IPython & Pandas: List Variables in DataFrame by Type

This snippet will give you a nice bulleted list of the variables in your DataFrame, organized by type:

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/882630 2015-07-17T18:29:33Z 2015-09-13T22:02:17Z Installing Python 3 Alongside Python 2 on OS X (Yosemite)

This was actually pretty easy. Here’s what I did:

  1. Opened a new Terminal window that wasn’t in a virtualenv.
  2. Ran brew update
  3. Ran brew install python3
  4. cd ~/.virtualenvs (the folder where my virtualenvs are)
  5. virtualenv -p `which python3` data3 to create a new virtualenv using Python 3
  6. workon data3 (this is from virtualenvwrapper)
  7. pip install "ipython[all]"

Update: I've been trying out pyenv, which is forked from my much-beloved rbenv. It seems to work fine with virtualenv via pyenv-virtualenv, but I could not get pyenv-virtualenvwrapper to work. I'm using this now to run an updated version of Python 2 without messing with my system Python, and it seems to work fine.

]]>
Max Masnick
tag:protips.maxmasnick.com,2013:Post/882614 2015-07-17T17:44:47Z 2017-04-07T12:26:42Z IPython and Jupyter Notebooks: Automatically Export .py and .html

[Updated 2016-03-04 to support Jupyter 4 notebooks – see below.]

IPython notebooks are stored in a format that is not particularly human-readable and doesn’t work well in version control.

One way to solve this problem is to automatically export the code from IPython notebooks into a vanilla Python file after each save.

It’s also useful to automatically generate a HTML file of the notebook on each save. This can be done manually in Jupyter (File > Download as > HTML), but if you always want this, doing it automatically is much easier.

Use the following code to automatically save a .py and a .html file when you save a notebook in Jupyter. These two files will be saved in the same folder as the parent .ipynb file.

First, run ipython locate profile default, which will give you the path to save the following code in.

Save the code below in this folder as ipython_notebook_config.py:

Now, run ipython notebook. You will see an error message in the terminal if there are any syntax or runtime errors with ipython_notebook_config.py. If everything looks good, go to your web browser, open a notebook, and click the Save/Checkpoint button (it’s the floppy disk icon in the Jupyter toolbar). You should see a .py and a .html file appear alongside your .ipynb file.

Update for Jupyter 4 notebooks

After the big split between IPython and Jupyter, and the accompanying update to Jupyter 4.x, the config file above no longer works. Instead, put the following in a file saved at ~/.jupyter/jupyter_notebook_config.py to achieve the same thing:

This code could be DRYer, but it does work.

]]>
Max Masnick