Better Selecting and Masking

by | 4 years ago

Making selections—quickly and accurately—is one of the most important skills to master in Photoshop, as it’s often the first step in working with layers, making masks, etc. In this article, we’ll look at effective methods for making selections and working with masks. Along the way, we’ll explore some very important “secrets” that can make things easier.

The Selection Tools and Commands

Although we’re not going to cover all of the following in detail, Photoshop has many selection tools and commands, each with their own advantages, and each designed for different selection situations.

Shape Selection Tools

  • Rectangular Marquee—for selecting rectangles and squares
  • Elliptical Marquee—for selecting ovals and circles
  • Single Column Marquee—for a one-pixel column selection
  • Single Row Marquee—for a one-pixel row selection
  • Polygonal Lasso—for selecting straight-edged shapes
  • Lasso—for selecting free-form shapes
  • Pen—selecting using anchor points and paths

Automated Selection Tools and Commands

  • Magic Wand—automatic selection based on a range of colors
  • Magnetic Lasso—senses edges to help make a selection
  • Color Range—selects by ranges of colors, with options to adjust
  • Quick Selection—the “edge detection” tool (click-and-drag)
  • Focus Area—makes a selection based on depth of field

Other Selection Methods (Found Under the Select Menu)

  • Quick Mask—view and edit the selection as a colored overlay
  • Load Selections from other sources—use layers or channels to create selections
  • Grow—expand an existing selection
  • Similar—select pixels similar to existing selection
  • Transform Selection—adjust a selection using transformation handles
  • Modify commands—a series of commands to adjust a selection
  • Save Selection—a means to “store” a selection for later use
  • Select and Mask—a workspace to refine and adjust the selection

My Top 10

Here are my top techniques when working with selections and masks.

  1. End up with a Great Selection: Although you may sometimes be able to make a great selection with one tool, chances are it’s going to take several steps and tools. As we know, there are many options for making selections both in tools and under menus, and it’s very likely that you’ll end up using multiple functions together.

For me, it’s always made better sense to think, “End up with a great selection,” rather than “Select something.”

If you’ve made an initial selection and it’s not quite right, you can use the same selection tool (or a different one) to tweak the selection. Think about fine-tuning your selection using keyboard modifiers and the Transform Selection command, in this way:

  • Hold down Shift and click-and-drag to add to the existing selection.
  • Hold down Option (PC: Alt) and click-and-drag to remove from an existing selection

As an example, start with a selection made with the Quick Selection tool (W) and then switch to the Lasso tool (L) to add the missing areas (with the Shift key held down).

You can also use Select>Transform Selection to click-and-drag on handles to resize a selection to fit the shape you need. Use the same techniques as you would with Free Transform (Command-T ), such as Shift to constrain and Command (PC: Ctrl) to edit the individual corners.

Once you’re in Transform Selection, you can also click the Warp icon (the warped grid shape with a curved arrow below it in the Options Bar) and drag the grid to adjust your selection.

  1. Something is Better than Nothing: This “philosophy” is similar to the concept of “End up with a great selection.” I sometimes see people trying an automated tool or method and being unhappy with the results because it wasn’t perfect—and then starting over from scratch. I prefer to think of this as a great start, compared to the alternative of my making a manual selection.

For example, the automated selection tools will sometimes work very well, but often make a selection that’s less than ideal; but rather than starting over again, consider whether you can build on the selection (as we discussed in the first tip).

The Magic Wand, Select>Color Range command, and the Quick Selection tools all make a selection more automatically than any other methods, and as a result, provide a fast way to get a great start.

BONUS: Change Lasso tools on the FlyUse this method to change between the Lasso, Magnetic Lasso, and Polygonal Lasso tool “on the fly,” without stopping and starting again to change tools. With the Magnetic Lasso (press Shift-L until it’s active in the Toolbar) start dragging out your selection. Without letting go of the mouse button, press-and-hold Option (PC: Alt). If you continue to drag, you’ll now be using the Lasso tool. Then, release the mouse button (but continue to hold Option ), and it will switch to the Polygonal Lasso tool. Click to add straight lines to the selection. Before you click to draw your last straight line, let go of Option (PC: Alt), click to draw that final line, and you’ll be back to the Magnetic Lasso. It takes a bit of practice but it’s a great timesaver.

  1. See What You’re Doing: One of the challenges of making selections is being able to tell how accurate your selection really is because sometimes the selection edges (a.k.a. marching ants) make it difficult to see how well you’re doing. For these situations, try using Quick Mask. With an active selection, click on the Quick Mask icon near the bottom of the Toolbar, or tap Q on your keyboard. The selection displays with a colored overlay that represents the areas that aren’t selected. The selected areas have no colored overlay. (Note: If your selected area is red, double-click the Quick Mask icon in the Toolbar to open the Quick Mask Options, and make sure Color Indicates is set to Masked Areas.)

To fine-tune the selection in Quick Mask, paint with black to add to the colored overlay and remove from the selection. To add to the selection, paint with white. You can use any method of “painting,” from using the Brush tool (B) to making a selection and filling it with black or white.

Hint 1: If you use the Brush tool, the size and edge of the brush will have an influence on the selection. Use smaller brushes for detailed areas; hard-edged brushes to create non-feathered selections; and soft-edged brushes to create feathered edges.

Hint 2: Use a series of short brushstrokes, rather than one long brushstroke. That way if you make a mistake, it’s easy to undo the last short stroke rather than all of the painting you’ve done.

When I think a selection is good enough to move on to the next step, it’s become a habit for me to always tap Q for Quick Mask to make sure I didn’t miss anything.

Tip: If the item you selected is red and the red overlay is too confusing to work with, you can double-click on the Quick Mask icon to change the color in the Quick Mask Options.

  1. Get Temporary Help: If you look at a photo and you can’t see much contrast between the subject and the background, consider this: If you can’t see the edge, Photoshop probably can’t either. In these cases, you can take advantage of the option to temporarily adjust the photo either using an adjustment layer or a Camera Raw smart object. In this image, the Quick Selection tool wouldn’t have much success since the edge is hard to detect.

By adding an adjustment layer such as Levels using the Layer>New Adjustment Layer menu (not the Image>Adjustments menu), we can temporarily over-adjust to help make the selection. Don’t worry about “ruining” the photo; all we care about is temporarily making it easier to see the edge.

Next, click on the photo layer in the Layers panel to make it active and make the selection.

Once the selection is complete and you’ve created a layer mask, throw away the Adjustment Layer.

You can use a similar process with RAW files, thanks to Camera Raw smart objects. To set this up, you have to open a RAW file in Camera Raw and click on the “link” at the bottom of the window to open the Workflow Options dialog. In that dialog, click on the checkbox to Open in Photoshop as Smart Objects, and click OK. The Open Image button should now say Open Object. Click that button to open the image in Photoshop.

This creates a “two-way editing street” between Camera Raw and Photoshop that lets you continue to make RAW adjustments from within Photoshop. In Photoshop, double-click on the layer thumbnail to return to Camera Raw and make editing changes.

In this example, the photo was deliberately shot to be dark to make it hard to distinguish between the subject’s hair and the background, which also means that it will be next to impossible to make an accurate selection. We want to select the subject and apply a layer mask to reveal a new background behind him.

—CREDIT: ©Dave Cross

After placing the subject’s layer on the new background (more on that in the next tip), jump back to Camera Raw and use the Basic panel to temporarily make it easier to see the edge of the subject’s hair. Click OK to return to Photoshop.

Once the selection is made and a layer mask is created, return to Camera Raw to make your final edits in the context of the new background. Click OK to return to Photoshop.

Remember, the key to both of these methods is to make temporary adjustments so that you can return to the original photo once the mask has been created.

  1. Work in Context: If the reason for your selection/mask is to put your subject on a different background image, this is perhaps the most important tip: Drag the subject onto the new background before making a selection. That way, every decision you make is in the context of the background, and you’ll avoid spending too much time making a selection and mask, only to find out that on the new background some of the selection work you did wasn’t necessary.
  2. Select in Pieces: This technique has saved me so much time! When I used to use Refine Edge (now Select and Mask), I tried to select both hard and soft edges at the same time and it never really worked very well. Now I make my selection in pieces.

After placing a Camera Raw smart object, I duplicate the layer, giving me two exact copies of the same photo.

Then I hide one of the layers by clicking on its Eye icon in the Layers panel and make a selection of areas with “hard” edges—in this case, the subject’s body. I deliberately don’t include her hair in the selection.

Then you can work on the other layer, making a selection of just the hair. That way in Select and Mask, you can use a much higher Radius because you don’t have to worry about having a negative impact on the hard edges of the subject (more on Select and Mask later).

You’ll end up with two layers, each with a layer mask to reveal the background below. Since the layers are linked Camera Raw smart objects, you can edit the settings of both layers at the same time: Double-click on one of the smart object thumbnails, make your changes, and both layers will update. Just remember than you’ll have to select both layers if you want to reposition your subject.

BONUS: A Second Chance—Once a layer mask has been created, you have a second chance to access the same controls to fine-tune the mask. To access these controls, double-click on the layer mask. The first time you do this, you’ll be asked whether you want to view the Properties panel or enter the Select and Mask workspace (whatever you select is the way it will behave moving forward). Select Properties. Once in the Properties panel, you can use the same methods as you use in Select and Mask to tweak your layer mask (it will update the existing mask rather than create a new one).

  1. The Select and Mask Workspace: Formerly known as Refine Edge, the Select and Mask workspace takes it up a notch with the addition of selection tools. This means that although you want to start off with as accurate a selection as possible, you can adjust your selection inside the Select and Mask workspace using the Quick Selection and Lasso tools. Once you have a selection, click on the Select and Mask button in the Options Bar.

You can change the View Mode in the Properties panel to help you refine your selection. The three views I think are most useful are:

  • Onion Skin (O): You can use the Transparency slider to alter how much of the original background (outside your selection) that you see.
  • On Layers (Y): Lets you see the layer you’re working on and the layer(s) below. (Remember “Work in Context” above?)
  • Black & White (K): This view is like a preview of the mask that will ultimately be created.

As a general guideline for selecting hair, start with a selection that’s slightly inside the edge of the hair. Then use a brush size for the Refine Edge Brush tool (R) that spans the existing selection and the hair you’re trying to select. Brush around the edge to select more hair: Remember that “something is better than nothing,” so even if it doesn’t look like you’ve selected all the fine hairs, if you’re in Black & White view and you can see something, that means you can refine it later (more on that next).

For many people, one of the biggest challenges with the Select and Mask workspace is trying to find the ideal value for the Edge Detection Radius; but if you use the suggestion earlier in this article about selecting in pieces, it’s no longer necessary to try to get both hard and soft edges in one mask.

And be sure to try the Decontaminate Colors checkbox in the Output settings, as this can often make a great improvement by removing any slight fringe that you see.

  1. The Magic of Overlay: When painting on a layer mask, changing your brush to Overlay mode can save you a ton of time. It’s particularly useful when you have areas of a mask that should be white but are light gray, or areas that should be black but are dark gray.

Let me remind you again of tip #2: Something is better than nothing. Rather than attempting to get a perfect selection of hair directly from Select and Mask, you can fix them by painting in Overlay mode if you at least have some areas of gray. Overlay mode changes the behavior of the Brush tool (B) so that white can’t paint over black, and black can’t paint over white, making it easier to adjust the gray areas.

To view the mask, hold down Option (PC: Alt) and click on the mask thumbnail in the Layers panel. Look for light gray areas that should be white.

Change the settings of the Brush tool to Overlay mode and 40–60% Opacity in the Options Bar. Press D to set your Foreground color to white and paint on the mask.

Once again, hold down Option (PC: Alt) and click on the mask thumbnail to return to the image view. This will let you continue painting on the mask while viewing the context of the subject on the new background. Paint over areas where you want to see more detail in those fine hairs. Don’t paint too much over the same area, as it can quickly look artificial.

If you have areas that are dark gray that should be black, switch the Foreground color to black and paint over them, once again with the brush set to Overlay mode.

  1. Automatic Masks: Use the Paste Into command to create an unlinked layer mask from a selection, making it easy to reposition the layer within the mask. Here’s an example:

To create a mock-up of a website on a tablet, make a selection of the tablet screen.

Then make a selection of the graphic that you want to paste, and copy it (Command-C ). Note: If the graphic is made up of multiple layers, use Edit>Copy Merged to make a merged copy of all the visible layers.

Switch to the tablet document with the active selection and choose Edit>Paste Special>Paste Into. This will create a new layer with a mask, and the layer and mask will be unlinked (so you can position and/or transform the photo within the mask).

Now, use Free Transform (Command-T ) with the Command (PC: Ctrl) key held down to transform each corner so the photo fits the screen. Press Enter.

Often we can add to the believability of projects like this by adding some finishing touches—in this case, a highlight. To add a highlight, add a layer above the pasted image, and with a large, soft-edged brush, paint with white. Then hold the Option (PC: Alt) key and click between the highlight layer and pasted-image layer to create a clipping mask and lower the opacity.

  1. Channels: For challenging selections, you can often take the approach of starting in the Channels panel with the goal of creating the equivalent of a mask. The ideal mask for replacing the sky in a photo like this would be a black “silhouette” of the trees with a white sky. Go to the Channels panel (Window>Channels) and look at the Red, Green, and Blue channels. Pick the one that gives you the best start toward the black-and-white silhouette (often the best channel will be the opposite of what you need).

Click on that channel (here it’s Blue) and drag it down to the Create New Channel icon to duplicate it.

It’s vital to make sure that you’re now working on the channel with “copy” at the end of its name of whichever channel you chose. To make the dark-gray areas darker, and the light-gray areas lighter, try using Levels. Just be careful not to over-adjust to the point where fine detail is lost. You can also paint in Overlay mode as described earlier.

Sometimes it’s helpful to invert the channel by pressing Command-I (PC: Ctrl-I) so you can spot issues you might have missed, and of course, if that’s the way you need the mask.

When the channel looks like the mask you need, hold down Command (PC: Ctrl) and click on the channel to load it as the selection. Click back on the RGB channel in the Channels panel.

For this example, back in the Layers panel, I made a mask from the selection and put a new sky photo below the layer.

BONUS: Useful Shortcuts

Here are a couple of shortcuts to add to your Photoshop skills:

  • Press X to swap the Foreground and Background color (this is incredibly useful so you can switch between painting with black and white).
  • Press the Left Bracket key () to increase the brush size.
  • Press the first number of the Opacity you want to use for your brush (e.g., 2 for 20%, 4 for 40%, etc.). Press 0 to bring the brush’s Opacity back to 100%.
  • Shift-click on the layer mask thumbnail to hide the mask; Shift-click again to show the effect of the mask.
  • To switch your brush to Overlay mode using only the keyboard, press Shift-Option-O (PC: Shift-Alt-O). To set the brush back to Normal blend mode, press Shift-Option-N (PC: Shift-Alt-N).