mm: munlock: manual pte walk in fast path instead of follow_page_mask()

mm: munlock: manual pte walk in fast path instead of follow_page_mask()

Currently munlock_vma_pages_range() calls follow_page_mask() to obtain
each individual struct page.  This entails repeated full page table
translations and page table lock taken for each page separately.

This patch avoids the costly follow_page_mask() where possible, by
iterating over ptes within single pmd under single page table lock.  The
first pte is obtained by get_locked_pte() for non-THP page acquired by the
initial follow_page_mask().  The rest of the on-stack pagevec for munlock
is filled up using pte_walk as long as pte_present() and vm_normal_page()
are sufficient to obtain the struct page.

After this patch, a 14% speedup was measured for munlocking a 56GB large
memory area with THP disabled.

Signed-off-by: Vlastimil Babka <redacted>
Cc: Jörn Engel <redacted>
Cc: Mel Gorman <redacted>
Cc: Michel Lespinasse <redacted>
Cc: Hugh Dickins <redacted>
Cc: Rik van Riel <redacted>
Cc: Johannes Weiner <redacted>
Cc: Michal Hocko <redacted>
Cc: Vlastimil Babka <redacted>
Signed-off-by: Andrew Morton <redacted>
Signed-off-by: Linus Torvalds <redacted>