Regarding the validity of a UTF-8 string when using the /u pattern modifier, some things to be aware of;
1. If the pattern itself contains an invalid UTF-8 character, you get an error (as mentioned in the docs above - "UTF-8 validity of the pattern is checked since PHP 4.3.5"
2. When the subject string contains invalid UTF-8 sequences / codepoints, it basically result in a "quiet death" for the preg_* functions, where nothing is matched but without indication that the string is invalid UTF-8
3. PCRE regards five and six octet UTF-8 character sequences as valid (both in patterns and the subject string) but these are not supported in Unicode ( see section 5.9 "Character Encoding" of the "Secure Programming for Linux and Unix HOWTO" - can be found at http://www.tldp.org/ and other places )
4. For an example algorithm in PHP which tests the validity of a UTF-8 string (and discards five / six octet sequences) head to: http://hsivonen.iki.fi/php-utf8/
The following script should give you an idea of what works and what doesn't;
<?php
$examples = array(
'Valid ASCII' => "a",
'Valid 2 Octet Sequence' => "\xc3\xb1",
'Invalid 2 Octet Sequence' => "\xc3\x28",
'Invalid Sequence Identifier' => "\xa0\xa1",
'Valid 3 Octet Sequence' => "\xe2\x82\xa1",
'Invalid 3 Octet Sequence (in 2nd Octet)' => "\xe2\x28\xa1",
'Invalid 3 Octet Sequence (in 3rd Octet)' => "\xe2\x82\x28",
'Valid 4 Octet Sequence' => "\xf0\x90\x8c\xbc",
'Invalid 4 Octet Sequence (in 2nd Octet)' => "\xf0\x28\x8c\xbc",
'Invalid 4 Octet Sequence (in 3rd Octet)' => "\xf0\x90\x28\xbc",
'Invalid 4 Octet Sequence (in 4th Octet)' => "\xf0\x28\x8c\x28",
'Valid 5 Octet Sequence (but not Unicode!)' => "\xf8\xa1\xa1\xa1\xa1",
'Valid 6 Octet Sequence (but not Unicode!)' => "\xfc\xa1\xa1\xa1\xa1\xa1",
);
echo "++Invalid UTF-8 in pattern\n";
foreach ( $examples as $name => $str ) {
echo "$name\n";
preg_match("/".$str."/u",'Testing');
}
echo "++ preg_match() examples\n";
foreach ( $examples as $name => $str ) {
preg_match("/\xf8\xa1\xa1\xa1\xa1/u", $str, $ar);
echo "$name: ";
if ( count($ar) == 0 ) {
echo "Matched nothing!\n";
} else {
echo "Matched {$ar[0]}\n";
}
}
echo "++ preg_match_all() examples\n";
foreach ( $examples as $name => $str ) {
preg_match_all('/./u', $str, $ar);
echo "$name: ";
$num_utf8_chars = count($ar[0]);
if ( $num_utf8_chars == 0 ) {
echo "Matched nothing!\n";
} else {
echo "Matched $num_utf8_chars character\n";
}
}
?>
패턴 변경자
아래 목록은 현재 존재하는 PCRE 변경자입니다. 괄호 안의 이름은 각 변경자에 대한 PCRE 내부의 이름입니다. 공백과 줄바꿈은 변경자에서 무시되며, 다른 문자는 오류를 발생합니다.
- i (PCRE_CASELESS)
- 이 변경자를 지정하면, 패턴의 문자는 대문자와 소문자를 구별하지 않습니다.
- m (PCRE_MULTILINE)
- 기본적으로, PCRE는 주어진 문자열을 하나의 "줄"로 취급합니다. (실제로 몇개의 라인을 가지더라도) "줄 시작" 메타문자(^)는 문자열의 처음만을 인식하며, "줄 끝" 메타문자($)는 문자열의 끝이나 (D 변경자가 지정되지 않는 한) 마지막 뉴라인의 직전만을 인식합니다. 이는 펄과 같습니다. 이 변경자를 지정하면, "줄 시작"과 "줄 끝"은 주어진 문자열의 모든 뉴라인 직후와 직전을 인식합니다. respectively, as well as at the very start and end. 이는 펄의 /m 변경자와 동일합니다. 주어진 문자열에 "\n" 문자가 존재하지 않거나 ^나 $ 패턴이 일어나지 않으면 이 변경자는 아무런 효과가 없습니다.
- s (PCRE_DOTALL)
- 이 변경자가 지정되면, 패턴의 점 메타문자는 뉴라인을 포함하는 모든 문자를 인식합니다. 지정하지 않으면, 뉴라인은 제외됩니다. 이 변경자는 펄의 /s 변경자와 동일합니다. [^a]와 같은 부정클래스는 이 변경자에 관계 없이 항상 뉴라인 문자를 포함합니다.
- x (PCRE_EXTENDED)
- 이 변경자가 지정되면, 공백 문자는 이스케이프 되거나 문자 클래스 안에 있을 경우를 제외하고, 완전히 무시합니다. 문자 클래스 밖에서 이스케이프 되지 않은 # 사이와 뉴라인 문자 다음의 문자도 무시합니다. 이는 펄의 /x 변경자와 같고, 복잡한 패턴 안에 코멘트를 사용할 수 있게 합니다. 그러나 이는 데이터 문자에만 해당하는 점에 주의하십시오. 공백 문자는 패턴의 특별한 문자 시퀀스 안에는 존재할 수 없습니다. 예를 들면, 조건 서브 패턴을 나타내는 (?( 시퀀스에는 나와서는 안됩니다.
- e (PCRE_REPLACE_EVAL)
- 이 변경자를 지정하면, preg_replace()는 변경할 문자열을 PHP 코드로 처리하고, 그 결과를 검색된 문자열의 이용하여 일반적인 치환을 합니다. 작은 따옴표, 큰 따옴표, 백슬래시와 NULL 문자는 백슬래시로 이스케이프됩니다.
preg_replace()만 이 변경자를 사용합니다; 다른 PCRE 함수는 무시합니다.
Note: 이 변경자는 PHP 3에서는 사용할 수 없습니다.
- A (PCRE_ANCHORED)
- 이 변경자를 지정하면, 패턴을 강제적으로 "고정"합니다. 이는 ("주어진 문자열"에서) 검색된 문자열의 시작에만 매치도록 강제합니다. 패턴 자체에서 특정한 구조를 가지게 하는, 펄에서는 유일한 방법으로 같은 효과를 얻을 수 있습니다.
- D (PCRE_DOLLAR_ENDONLY)
- 이 변경자가 설정되면, 패턴의 달러($) 메타문자는 주어진 문자열의 마지막에만 대응합니다. 이 변경자 없이는, 달러는 마지막 문자가 뉴라인일 경우에는 바로 직전의 문자에도 매칭합니다. (마지막이 아닌 뉴라인은 제외합니다) 이 변경자는 m 변경자가 지정되었을때는 무시됩니다. 펄에는 이 변경자가 존재하지 않습니다.
- S
- 패턴이 여러번 이용되면, 매칭에 걸리는 시간을 절약하기 위해서 분석에 더 많은 시간을 들일 가치가 있습니다. 이 변경자를 지정하면, 추가 분석을 행합니다. 현 시점에서, 패턴의 분석은 하나의 고정된 시작 문자를 가지지 않는 비고정 패턴에만 유용합니다.
- U (PCRE_UNGREEDY)
- 이 변경자는 수량 지시의 "greediness"를 뒤집습니다. 그리하여 기본값으로 not greedy하게 합니다. 하지만 "?"가 붙으면 greedy하게 됩니다. 이는 펄과 호환되지 않습니다. 패턴 안에서 변경자 설정으로 (?U)처럼 지정하거나, 수량지시어 뒤의 물음표로 지정할 수 있습니다. (예. .*?)
- X (PCRE_EXTRA)
- 이 변경자는 펄과 호환되지 않는 PCRE의 추가 기능을 사용하게 합니다. 패턴의 문자와 결합된 백슬래시가 특별한 의미를 지니지 않을 경우에 에러를 발생시켜서, 차후에 추가 기능을 위해 예약해둡니다. 기본적으로 펄은, 문자와 결합된 백슬래시가 특별한 의미를 지니지 않을 경우에는 글자로 취급합니다. 이 변경자는 다른 기능을 제어하지 않습니다.
- J (PCRE_INFO_JCHANGED)
- 내부 옵션 (?J) 설정은 영역의 PCRE_DUPNAMES 옵션을 변경합니다. 서브패턴에 동일한 이름을 허용합니다.
- u (PCRE_UTF8)
- 이 변경자는 펄과 호환되지 않는 PCRE의 추가 기능을 사용하게 합니다. 패턴 문자열을 UTF-8으로 취급합니다. 유닉스에서는 PHP 4.1.0부터, win32에서는 PHP 4.2.3부터 사용할 수 있습니다. PHP 4.3.5부터 패턴의 UTF-8 유효성이 검사됩니다.
add a note
User Contributed Notes 12 notes
hfuecks at nospam dot org ¶
18 years ago
varrah NO_GARBAGE_OR_SPAM AT mail DOT ru ¶
18 years ago
Spent a few days, trying to understand how to create a pattern for Unicode chars, using the hex codes. Finally made it, after reading several manuals, that weren't giving any practical PHP-valid examples. So here's one of them:
For example we would like to search for Japanese-standard circled numbers 1-9 (Unicode codes are 0x2460-0x2468) in order to make it through the hex-codes the following call should be used:
preg_match('/[\x{2460}-\x{2468}]/u', $str);
Here $str is a haystack string
\x{hex} - is an UTF-8 hex char-code
and /u is used for identifying the class as a class of Unicode chars.
Hope, it'll be useful.
phpman at crustynet dot org dot uk ¶
13 years ago
The description of the "u" flag is a bit misleading. It suggests that it is only required if the pattern contains UTF-8 characters, when in fact it is required if either the pattern or the subject contain UTF-8. Without it, I was having problems with preg_match_all returning invalid multibyte characters when given a UTF-8 subject string.
It's fairly clear if you read the documentation for libpcre:
In order process UTF-8 strings, you must build PCRE to include UTF-8
support in the code, and, in addition, you must call pcre_compile()
with the PCRE_UTF8 option flag, or the pattern must start with the
sequence (*UTF8). When either of these is the case, both the pattern
and any subject strings that are matched against it are treated as
UTF-8 strings instead of strings of 1-byte characters.
[from http://www.pcre.org/pcre.txt]
Anonymous ¶
4 years ago
A warning about the /i modifier and POSIX character classes:
If you're using POSIX character classes in your regex that indicate case such as [:upper:] or [:lower:] in combination with the /i modifier, then in PHP < 7.3 the /i modifier will take precedence and effectively make both those character classes work as [:alpha:], but in PHP >= 7.3 the character classes overrule the /i modifier.
arash dot dalir at gmail dot com ¶
6 years ago
the PCRE_INFO_JCHANGED modifier is apparently not accepted as a global option (after the closing delimiter) in PHP versions <= 5.4 (not checked in PHP 5.5) but allowed in PHP 5.6 (also not checked in PHP 7.X)
The following pattern doesn't work in PHP 5.4, but it works in PHP 5.6:
<?php
//test.php
preg_match_all('/(?<dup_name>\d{1,4})\-(?<dup_name>\d{1,2})/J', '1234-23', $matches);
var_dump($matches);
/*
output in PHP 5.4:
Warning: preg_match_all(): Unknown modifier 'J' in test.php on line 3
NULL
--------------
output PHP 5.6:
array(4) {
[0]=> array(1) { [0]=> string(7) "1234-23" }
["dup_name"]=> array(1) { [0]=> string(2) "23" }
[1]=> array(1) { [0]=> string(4) "1234" }
[2]=> array(1) { [0]=> string(2) "23" }
}
*/
?>
in order to resolve this issue in PHP 5.4, one can use the (?J) pattern modifier, which indicates the pattern (from that point forward) allows duplicate names for subpatterns.
code which works in PHP 5.4:
<?php
preg_match_all('/(?J)(?<dup_name>\d{1,4})\-(?<dup_name>\d{1,2})/', '1234-23', $matches);
var_dump($matches);
/*
output in PHP 5.4:
array(4) {
[0]=> array(1) { [0]=> string(7) "1234-23" }
["dup_name"]=> array(1) { [0]=> string(2) "23" }
[1]=> array(1) { [0]=> string(4) "1234" }
[2]=> array(1) { [0]=> string(2) "23" }
}
--------------
output in PHP 5.6 (the same as with /J):
array(4) {
[0]=> array(1) { [0]=> string(7) "1234-23" }
["dup_name"]=> array(1) { [0]=> string(2) "23" }
[1]=> array(1) { [0]=> string(4) "1234" }
[2]=> array(1) { [0]=> string(2) "23" }
}
*/
?>
Daniel Klein ¶
12 years ago
If the _subject_ contains utf-8 sequences the 'u' modifier should be set, otherwise a pattern such as /./ could match a utf-8 sequence as two to four individual ASCII characters. It is not a requirement, however, as you may have a need to break apart utf-8 sequences into single bytes. Most of the time, though, if you're working with utf-8 strings you should use the 'u' modifier.
If the subject doesn't contain any utf-8 sequences (i.e. characters in the range 0x00-0x7F only) but the pattern does, as far as I can work out, setting the 'u' modifier would have no effect on the result.
michal dot kocarek at brainbox dot cz ¶
14 years ago
In case you're wondering, what is the meaning of "S" modifier, this paragraph might be useful:
When "S" modifier is set, PHP calls the pcre_study() function from the PCRE API before executing the regexp. Result from the function is passed directly to pcre_exec().
For more information about pcre_study() and "Studying the pattern" check the PCRE manual on http://www.pcre.org/pcre.txt
PS: Note that function names "pcre_study" and "pcre_exec" used here refer to PCRE library functions written in C language and not to any PHP functions.
Hayley Watson ¶
3 years ago
Starting from 7.3.0, the 'S' modifier has no effect; this analysis is now always done by the PCRE engine.
ebarnard at marathonmultimedia dot com ¶
17 years ago
When adding comments with the /x modifier, don't use the pattern delimiter in the comments. It may not be ignored in the comments area. Example:
<?php
$target = 'some text';
if(preg_match('/
e # Comments here
/x',$target)) {
print "Target 1 hit.\n";
}
if(preg_match('/
e # /Comments here with slash
/x',$target)) {
print "Target 1 hit.\n";
}
?>
prints "Target 1 hit." but then generates a PHP warning message for the second preg_match():
Warning: preg_match() [function.preg-match]: Unknown modifier 'C' in /ebarnard/x-modifier.php on line 11
Wirek ¶
6 years ago
A hint for those of you who are trying to fight off (or work around at least) the problem of matching a pattern correctly at the end ($) of any line in multiple lines mode (/m).
<?php
// Various OS-es have various end line (a.k.a line break) chars:
// - Windows uses CR+LF (\r\n);
// - Linux LF (\n);
// - OSX CR (\r).
// And that's why single dollar meta assertion ($) sometimes fails with multiline modifier (/m) mode - possible bug in PHP 5.3.8 or just a "feature"(?).
$str="ABC ABC\n\n123 123\r\ndef def\rnop nop\r\n890 890\nQRS QRS\r\r~-_ ~-_";
// C 3 p 0 _
$pat1='/\w$/mi'; // This works excellent in JavaScript (Firefox 7.0.1+)
$pat2='/\w\r?$/mi';
$pat3='/\w\R?$/mi'; // Somehow disappointing according to php.net and pcre.org
$pat4='/\w\v?$/mi';
$pat5='/(*ANYCRLF)\w$/mi'; // Excellent but undocumented on php.net at the moment
$n=preg_match_all($pat1, $str, $m1);
$o=preg_match_all($pat2, $str, $m2);
$p=preg_match_all($pat3, $str, $m3);
$r=preg_match_all($pat4, $str, $m4);
$s=preg_match_all($pat5, $str, $m5);
echo $str."\n1 !!! $pat1 ($n): ".print_r($m1[0], true)
."\n2 !!! $pat2 ($o): ".print_r($m2[0], true)
."\n3 !!! $pat3 ($p): ".print_r($m3[0], true)
."\n4 !!! $pat4 ($r): ".print_r($m4[0], true)
."\n5 !!! $pat5 ($s): ".print_r($m5[0], true);
// Note the difference among the three very helpful escape sequences in $pat2 (\r), $pat3 (\R), $pat4 (\v) and altered newline option in $pat5 ((*ANYCRLF)) - for some applications at least.
/* The code above results in the following output:
ABC ABC
123 123
def def
nop nop
890 890
QRS QRS
~-_ ~-_
1 !!! /\w$/mi (3): Array
(
[0] => C
[1] => 0
[2] => _
)
2 !!! /\w\r?$/mi (5): Array
(
[0] => C
[1] => 3
[2] => p
[3] => 0
[4] => _
)
3 !!! /\w\R?$/mi (5): Array
(
[0] => C
[1] => 3
[2] => p
[3] => 0
[4] => _
)
4 !!! /\w\v?$/mi (5): Array
(
[0] => C
[1] => 3
[2] => p
[3] => 0
[4] => _
)
5 !!! /(*ANYCRLF)\w$/mi (7): Array
(
[0] => C
[1] => 3
[2] => f
[3] => p
[4] => 0
[5] => S
[6] => _
)
*/
?>
Unfortunately, I haven't got any access to a server with the latest PHP version - my local PHP is 5.3.8 and my public host's PHP is version 5.2.17.
Wirek ¶
6 years ago
An important addendum (with new $pat3_2 utilising \R properly, its results and comments):
Note that there are (sometimes difficult to grasp at first glance) nuances of meaning and application of escape sequences like \r, \R and \v - none of them is perfect in all situations, but they are quite useful nevertheless. Some official PCRE control options and their changes come in handy too - unfortunately neither (*ANYCRLF), (*ANY) nor (*CRLF) is documented here on php.net at the moment (although they seem to be available for over 10 years and 5 months now), but they are described on Wikipedia ("Newline/linebreak options" at https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions) and official PCRE library site ("Newline convention" at http://www.pcre.org/original/doc/html/pcresyntax.html#SEC17) pretty well. The functionality of \R appears somehow disappointing (with default configuration of compile time option) according to php.net as well as official description ("Newline sequences" at https://www.pcre.org/original/doc/html/pcrepattern.html#newlineseq) when used improperly.
A hint for those of you who are trying to fight off (or work around at least) the problem of matching a pattern correctly at the end (or at the beginning) of any line even without the multiple lines mode (/m) or meta-character assertions ($ or ^).
<?php
// Various OS-es have various end line (a.k.a line break) chars:
// - Windows uses CR+LF (\r\n);
// - Linux LF (\n);
// - OSX CR (\r).
// And that's why single dollar meta assertion ($) sometimes fails with multiline modifier (/m) mode - possible bug in PHP 5.3.8 or just a "feature"(?) of default configuration option for meta-character assertions (^ and $) at compile time of PCRE.
$str="ABC ABC\n\n123 123\r\ndef def\rnop nop\r\n890 890\nQRS QRS\r\r~-_ ~-_";
// C 3 p 0 _
$pat3='/\w\R?$/mi'; // Somehow disappointing according to php.net and pcre.org when used improperly
$pat3_2='/\w(?=\R)/i'; // Much better with allowed lookahead assertion (just to detect without capture) without multiline (/m) mode; note that with alternative for end of string ((?=\R|$)) it would grab all 7 elements as expected, but '/(*ANYCRLF)\w$/mi' is more straightforward in use anyway
$p=preg_match_all($pat3, $str, $m3);
$r=preg_match_all($pat3_2, $str, $m4);
echo $str."\n3 !!! $pat3 ($p): ".print_r($m3[0], true)
."\n3_2 !!! $pat3_2 ($r): ".print_r($m4[0], true);
// Note the difference between the two very helpful escape sequences in $pat3 and $pat3_2 (\R) - for some applications at least.
/* The code above results in the following output:
ABC ABC
123 123
def def
nop nop
890 890
QRS QRS
~-_ ~-_
3 !!! /\w\R?$/mi (5): Array
(
[0] => C
[1] => 3
[2] => p
[3] => 0
[4] => _
)
3_2 !!! /\w(?=\R)/i (6): Array
(
[0] => C
[1] => 3
[2] => f
[3] => p
[4] => 0
[5] => S
)
*/
?>
Unfortunately, I haven't got any access to a server with the latest PHP version - my local PHP is 5.3.8 and my public host's PHP is version 5.2.17.
damian dot driscoll at gmail dot com ¶
6 years ago
The PCRE_INFO_JCHANGED modifier works in version 5.6.31 but not in 5.6.16, which generates an 'unknown modifier J' warning.
