NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries, and an active discussion forum.
Source: https://www.nltk.org/
Installation: https://www.nltk.org/install.html
The second aspect is summarised in that phrase “potential patterns”. How so? The process operates in two stages. First, all the effort of a concordancer or a word-listing application goes into reducing a vast and complex object to a much simpler shape. That is, a set of 100 million words on a confusing wealth of topics in a variety of styles and produced by innumerable people for a lot of different reasons gets reduced to a mere list in alphabetical order. A rich chaos of language is reduced, it is “boiled down” to a simpler set. In the vapours that have steamed off are all the facts about who wrote the texts and what they meant.We have therefore lost a great deal in that process, and if it damaged the original texts we would never dare do it.
The advantage comes in the second stage where one examines the boiled down extract, the list of words, the concordance. It is here that something not far different from the sometimes-scorned “intuition” comes in. This is imagination. Insight. Human beings are unable to see shapes, lists, displays, or sets without insight, without seeing in them “patterns”. It seems to be a characteristic of the homo sapiens mind that it is often unable to see things “as they are” but imposes on them a tendency, a trend, a pattern. From the earliest times, the very stars in the sky have been perceived as belonging in “constellations”. This capability can come at a cost, of course: it may be easy to spot a pattern in a cloud or in a constellation and thereby build up a mistaken theory; but the point is that it is this ordinary imaginative capacity, that of seeing a pattern, which is there in all of us and which makes it possible for corpus-based methods to make a relatively large impact on language theory. For with these twin resources, namely the tools to manipulate a lot of data in many different ways and without wasting much time, combined with the power of imagination and pattern-recognition, it becomes possible to chase up patterns that seem to be there and come up with insights affecting linguistic theory itself. The tools we use generate patterns (lists, plots, colour arrangements) and it is when we see these that in some cases the pattern “jumps out” at us. In other cases we may need training to see the patterns but the endeavour is itself largely a search for pattern.
Scott, M. & Tribble, C. 2006. Textual Patterns: keyword and corpus analysis in language education. Amsterdam: Benjamins. (pp. 5-6).
Online editor that can help you to improve word choice and writing style, and, optionally, help you to detect grammar mistakes and plagiarism. To check word count, simply place your cursor into the text box above and start typing. You’ll see the number of characters and words increase or decrease as you type, delete, and edit them. You can also copy and paste text from another program over into the online editor above. The Auto-Save feature will make sure you won’t lose any changes while editing, even if you leave the site and come back later.
—————————————————————————————————————————————————————————————————————
BBEdit / BBEdit-Lite / TextWrangler Regular Expression Guide Modified: 2018/08/10 01:19
—————————————————————————————————————————————————————————————————————
NOTES:
The PCRE engine (Perl Compatible Regular Expressions) is what BBEdit and TextWrangler use.
Items I’m unsure of are marked ‘# PCRE?’. The list while fairly comprehensive is not complete.
————————————————————————————————————————————————————————————————————
PATTERN MODIFIERS (switches)
————————————————————————————————————————————————————————————————————
i Case-insensitive
m Multiline : allow the grep engine to match at ^ and $ after and before at \r or \n.
s Magic Dot : allows . to match \r and \n
x Free-spacing: ignore unescaped white space; allow inline comments in grep patterns.
(?imsx) On
(?-imsx) Off
(?i-msx) Mixed
———————————————————————————————————————————————————————————————————
Regex Meta-Characters:
———————————————————————————————————————————————————————————————————
. Any character except newline or carriage return
[ ] Any single character of set
[^ ] Any single character NOT of set
—————————————————————————————————————————————————————————————————
Case-Change Operators
—————————————————————————————————————————————————————————————————
\E Change case – acts as an end delimiter to terminate runs of \L & \U.
\l Change case of only the first character to the right lower case. (Note: lowercase ‘L’)
\L Change case of all text to the right to lowercase.
\u Change case of only the first character to the right to uppercase.
\U Change case of all text to the right to uppercase.
——————————————————————————————————————————————————————————————
White-Space or Non-White-Space
——————————————————————————————————————————————————————————————
\t Tab
\n Linefeed
\r Return
\R Return or Linefeed or Windows CRLF (matches any Unicode newline sequence).
\f Formfeed
\s Whitespace character equivalent to [ \t\n\r\f]
\S Non-whitespace character
——————————————————————————————————————————————————————
\W Non-word character
\w Word character[0-9A-Za-z_]
\z End of a string
\Z End of a string, or before newline at the end
(?#) Comment
(?:) Grouping without backreferences
(?=) Zero-width positive look-ahead assertion
(?!) Zero-width negative look-ahead assertion
(?>) Nested anchored sub-regexp stops backtracking
(?imx-imx) Turns on/off imx options for rest of regexp
(?imx-imx:…) Turns on/off imx options, localized in group # ‘…’ indicates added regex pattern
———————————————————————————————————————————————————————————————
PERL-STYLE PATTERN EXTENSIONS : BBEdit Documentation : ‘…’ indicates added regex pattern
————————————————————————————————————————————————————————————————
Extension Meaning
————————————————————————————————————————————————————————————————
(?:…) Cluster-only parentheses, no capturing
(?#…) Comment, discard all text between the parentheses
(?imsx-imsx) Enable/disable pattern modifiers
(?imsx-imsx:…) Cluster-only parens with modifiers
(?=…) Positive lookahead assertion
(?!…) Negative lookahead assertion
(?<=…) Positive lookbehind assertion
(?…) Match non-backtracking subpattern (“once-only”)
(?R) Recursive pattern
—————————————————————————————————————————————————————————————————
POSITIONAL ASSERTIONS (duplicatation of above)
—————————————————————————————————————————————————————————————————
POSITIVE LOOKAHEAD ASSERTION: (?=’pattern’)
NEGATIVE LOOKAHEAD ASSERTION: (?!’pattern’)
POSITIVE LOOKBEHIND ASSERTION: (?<=’pattern’) # Lookbehind Assertions are of fixed-length
NEGATIVE LOOKBEHIND ASSERTION: (?<!’pattern’)
————————————————————————————————————————————————————————————————
SPECIAL CHARACTER CLASSES (POSIX standard except where ‘Perl Extension’ is indicated):
———————————————————————————————————————————————————————————————
CLASS MEANING
———————————————————————————————————————————————————————————————
[[:alnum:]] Alpha-numeric characters
[[:alpha:]] Alphabetic characters
[[:ascii:]] Character codes 0-127 # Perl Extension
[[:blank:]] Horizontal whitespace
[[:cntrl:]] Control characters
[[:digit:]] Decimal digits (same as \d)
[[:graph:]] Printing characters, excluding spaces
[[:lower:]] Lower case letters
[[:print:]] Printing characters, including spaces
[[:punct:]] Punctuation characters
[[:space:]] White space (same as \s)
[[:upper:]] Upper case letters
[[:word:]] Word characters (same as \w) # Perl Extension
[[:xdigit:]] Hexadecimal digits
Usage example of multiple character classes:
[[:alpha:][:digit:]]«Negated» character class example:
[[:^digit:]]+** POSIX-style character class names are case-sensitive
** The outermost brackets above indicate a RANGE; the class name itself looks like this: [:alnum:]
—————————————————————————————————————————————————————————————
CONDITIONAL SUBPATTERNS
—————————————————————————————————————————————————————————————
Conditional subpatterns allow you to apply “if-then” or “if-then-else” logic to pattern matching.
The “if” portion can either be an integer between 1 and 99, or an assertion.
The forms of syntax for an ordinary conditional subpattern are:
if-then: (?(condition)yes-pattern)if-then-else: (?(condition)yes-pattern|no-pattern)
and for a named conditional subpattern are:
if-then: (?P<NAME>(condition)yes-pattern)if-then-else: (?P(condition)yes-pattern|no-pattern)
If the condition evaluates as true, the “yes-pattern” portion attempts to match. Otherwise, the
“no-pattern” portion does (if there is a “no-pattern”).
———————————————————————————————————————————————————————————————
REVISION NOTES:
———————————————————————————————————————————————————————————————
2016/02/29 17:23
\G metacharacter added.
Tested with BBEdit 11.5.1 & TextWrangler 5.0.2.
Also available in ICU RegEx:
————————————————————————————————————————————————————————————————
Source: Melissa Bond, Olaf Zawacki-Richter & Mark Nichols. 2019. Revisiting five decades of educational technology research: A content and authorship analysis of the British Journal of Educational Technology. British Journal of Educational Technology, 50,1, 12–63.