20090529
http://www.terrencemiao.com/Webmail/msg00947.html
20090527
Posted by Tom Cunningham on February 5 2004 5:42am
1. Have a normalized versions of the important columns: where you've stripped punctuation and converted numerals to words ('1' to 'one'). Likewise normalise the search string.
2. Have a combined fulltext index on your searchable columns to use in your 'WHERE' clause, but then have separate fulltext indexes on each column to use in the 'ORDER BY' clause, so they can have different weights.
3. For the scoring algorithm, include the independent importance of that record, and include a match of the inclusive index against stemmed versions of the search words (as: "wedding" => "wed", "weddings").
4. If there's exactly one result, go straight to that record.
5. If you get no results, try matching against the start of the most important column (WHERE column LIKE 'term%'), and put a 5-character index on that column. This helps if someone is searching on a very short word or a stopword.
6. Reduce minimum word length to 3, and make a new stopwords list just using "a an and the is in which we you to on this by of with". Use "REPAIR TABLE xxx QUICK" to rebuild the index and make a note of the index-file (xxx.MYI) size before and after you make changes. Then use ft_dump to tune.
http://dev.mysql.com/doc/refman/5.1/en/fulltext-search.html
11.8. Full-Text Search Functions
[+/-]
MATCH ( col1,col2,...) AGAINST (expr [search_modifier])
search_modifier:
{
IN BOOLEAN MODE
| IN NATURAL LANGUAGE MODE
| IN NATURAL LANGUAGE MODE WITH QUERY EXPANSION
| WITH QUERY EXPANSION
}
MySQL has support for full-text indexing and searching:
A full-text index in MySQL is an index of type
FULLTEXT.Full-text indexes can be used only with
MyISAMtables, and can be created only forCHAR,VARCHAR, orTEXTcolumns.A
FULLTEXTindex definition can be given in theCREATE TABLEstatement when a table is created, or added later usingALTER TABLEorCREATE INDEX.For large data sets, it is much faster to load your data into a table that has no
FULLTEXTindex and then create the index after that, than to load data into a table that has an existingFULLTEXTindex.
Full-text searching is performed using MATCH() ... AGAINST syntax. MATCH() takes a comma-separated list that names the columns to be searched. AGAINST takes a string to search for, and an optional modifier that indicates what type of search to perform. The search string must be a literal string, not a variable or a column name. There are three types of full-text searches:
A boolean search interprets the search string using the rules of a special query language. The string contains the words to search for. It can also contain operators that specify requirements such that a word must be present or absent in matching rows, or that it should be weighted higher or lower than usual. Common words such as “some” or “then” are stopwords and do not match if present in the search string. The
IN BOOLEAN MODEmodifier specifies a boolean search. For more information, see Section 11.8.2, “Boolean Full-Text Searches”.A natural language search interprets the search string as a phrase in natural human language (a phrase in free text). There are no special operators. The stopword list applies. In addition, words that are present in 50% or more of the rows are considered common and do not match. Full-text searches are natural language searches if the
IN NATURAL LANGUAGE MODEmodifier is given or if no modifier is given.A query expansion search is a modification of a natural language search. The search string is used to perform a natural language search. Then words from the most relevant rows returned by the search are added to the search string and the search is done again. The query returns the rows from the second search. The
IN NATURAL LANGUAGE MODE WITH QUERY EXPANSIONorWITH QUERY EXPANSIONmodifier specifies a query expansion search. For more information, see Section 11.8.3, “Full-Text Searches with Query Expansion”.
The IN NATURAL LANGUAGE MODE and IN NATURAL LANGUAGE MODE WITH QUERY EXPANSION modifiers were added in MySQL 5.1.7.
Constraints on full-text searching are listed in Section 11.8.5, “Full-Text Restrictions”.
http://www.fastechws.com/tricks/sql/full-text-search-boolean.php
Full Text Searching with MySQL
When you want to let visitors search through a web-site's data by typing a search string, possibly containing multiple words, what's the best way to do that?
This is a question that web developers have been asking for a long time. There's many views on this subject, and many opinions. I generally want the best searching ability I can get without spending aeons of time (and cost) developing it.
Simple Searches
You may be thinking this is no big deal - your data is in a SQL database, so why not just do word-matching with "LIKE":
SELECT * FROM t1
WHERE description LIKE '%word%';
That is fine when you only have one word to search for, and one field to search in. But what if you have three fields (name, title, description) and the user types five keywords, such as "where is the Saharan Desert"? By eye, you can see that the really useful search keywords are "Saharan" and "Desert" - but how can your code possibly know that? If you require all 5 words to match, you won't return any matches at all.
Even if you knew that there are only 2 valid keywords, you can't just do a single match like this:
SELECT * FROM t1
WHERE description LIKE '%saharan%desert%';
That won't match everything that should match, because maybe some of the descriptions have the words in the other order. For two keywords, you really have to do two matches:
SELECT * FROM t1
WHERE description LIKE '%saharan%desert%'
OR description LIKE '%desert%saharan%';
You can imagine how this degrades - every combination of 3 keywords requires 6 different LIKE's; 4 keywords requires 24 combinations - that's factorial growth; it starts to get really big, and really slow!
Database Solution - Full Text Search
A really good solution to this and other problems related to search are solved by Full Text Search. Let the Database automatically maintain a special index of some sort that helps it find your matches quickly! You just provide the keywords to search for and which table fields should be searched. Simple.
The basic form is called "Natural Language Search", and has a number of cool features, including the ability to have a "stop list" - a list of words that are so popular, we ignore them because it would return too many records (too many mis-matches). Words like "is", "the", and "and" are way too popular. MySQL's built-in stop list is a text file that the administrator (root user) can change if needed.
In fact, with MySQL, when it builds the FT index for the specified fields in a table, it figures out other stop words - any word used in 50% or more of the records is automatically excluded from matching - which is good, you generally don't want half your table's records coming back as positive matches.
FTS queries look like this in MySQL:
SELECT * FROM articles
WHERE MATCH (title,body) AGAINST ('what is the saharan desert');
Setting up Full Text Search
For that query to work, your table must already have a FTS index created, which must have the exact fields that you're matching against (title and body, in the above example).
To create that index you would do the following, one time only:
ALTER TABLE articles
ADD FULLTEXT (title,body);
If you ever need to match against a different set of fields - say, just title, or just body, you'll need to create a whole other FULLTEXT index just for that. You can have many indexes on any table; it just takes more space (disk space, and sometimes memory).
Fancier Method - Boolean Matching
You know those advanced features of search engines like Google, where you can specify negative keywords with "-" sign, as in:
chili -bar -"new york" |
because you want info on chili the food, not the bar and grill or the city in New York.
That, and a lot more, is built in to MySQL FTS - using Boolean Expressions. Boolean expressions include ways to vary the priority value based on certain keywords being there, or not being there. You can do partial-word matches (like "*" globbing in Unix, sort of). It honors double-quotes around "multiple words" when you want to find those words next to each other in the order shown. It can even let you group pieces of the boolean expression (with parentheses). Read the MySQL docs (see References below) to learn the syntax.
Notes and Limits with Full Text Search
Make sure your test data has at least 3 records. Think about it: 2 records won't work because ALL the words are found on 50% of the records, so everything's stoplisted!
If you want really advanced features such as configurable weight values per field, you will have to write more code yourself. You could put separate FT indexes on each field of the table, then build a query that has multiplier values (weights) for each field's results. This would probably result in more work for the database engine, but give you tighter results.
Another thing to remember is that (at least with MySQL), the Natural Language matching does not necessarily tell "how well" the match occurred; you only know that it matched or it didn't match. To get a nicer priority value back, use Boolean matching. You get a floating point value between 0 and 1, I believe.
References - MySQL Manual Online
MySQL Natural Language Matching
MySQL Boolean Expression Matching
Those links are for MySQL 5.0, but you can click on the other versions to see the related pages there.
Books
MySQL Cookbook from O'Reilly and Associates
Chapter 5 has a lot of info on Full Text Searches.
Conclusion
FTS has been around for a while - it should work with MySQL versions 4, 5, and the upcoming version 6. FTS exists in many other databases too, not just MySQL. However the syntax may vary with other database systems.
Overall, Full Text Search is very powerful and useful, and very easy to configure and use. It certainly saves a lot of time from having to do individual keyword matches yourself and try to determine which matches are "better" than which others.
<ul>1. Navigate to the directory where you extracted Universal_Customizer.zip to and open the 'U3CUSTOM' folder. <br>2. Copy your custom files* to that folder. <br>3. Go to the parent directory. <br>4. Execute 'ISOCreate.cmd' (It will create an ISO with the CD name of 'U3CDROM') <br>5. Launch 'Universal_Customizer' and your done.</ul><br>*Use the files in the folder where you backed up your U3 CD-ROM if you want to restore your U3 LaunchPad. <br><br><span style="font-weight: bold; ">U3 FIRMWARE ISO's</span> <br><a href="http://www.sendspace.com/file/5vr8rd" target="_blank" class="postlink" style="color: rgb(0, 102, 153); text-decoration: none; ">Memorex LaunchPad</a> <br><a href="http://www.sendspace.com/file/clwcdg" target="_blank" class="postlink" style="color: rgb(0, 102, 153); text-decoration: none; ">SanDisk LaunchPad</a> <br>
http://www.wayne-robinson.com/journal/2006/11/11/multiple-values-from-scriptaculous-autocomplete.html
Wayne Robinson in javascript, programming, ruby on rails
Ever wanted to extract multiple values from a Script.aculo.us Google Suggest-like autocomplete text field? I recently did and here's how.
If you aren't aware of how to use autocomplete text fields, please read over the simple and customised demos available at Script.aculo.us. Also, a warning, this demo utilises Ruby on Rails however, with a little bit of modification, this should work using the Script.aculo.us library without Ruby on Rails.
This example will auto-populate a state and postcode based on the user's selected suburb (yes, I'm Australian).
The first step is to create a view for the page containing the autocomplete field and for the autocomplete field itself.
Controller:
def new
# This is the main controller that will
# contain the autocomplete field
@contact = Contact.new
end
def auto_complete_for_suburb
suburb = params[:suburb]
@surburbs = Suburb.find_by_name(suburb,
:order => "name", :limit => 20)
render :partial => "auto_complete_suburb"
end
View for the new action (assume an application.rhtml template has been created, this template must include the Prototype and Script.aculo.us javascript libraries):
<%= form_tag({:action => :create}, {:method => :post}) %>
Name:
<%= text_field(:contact, :name) %>
Suburb:
<%= text_field_with_autcomplete(:contact, :suburb,
:select => "value",
:after_update_element =>
"function (ele, value) {
$("contact_state").value =
Ajax.Autocompleter.extract_value(value,
'STATE');
$("contact_postcode").value =
Ajax.Autocompleter.extract_value(value,
'POSTCODE'); }
") %>
State:
<%= text_field(:contact, :state, :size => 10) %>
Postcode:
<%= text_field(:contact, :postcode,
:size => 10) %>
<%= end_form_tag %>
Before we continue any further, it is worth-while defining the _auto_complete_suburb.rhtml partial.
<% unless @suburbs.nil? -%>
<% @suburbs.each do | suburb | -%>
<%= h("#{suburb[:name]}, #{suburb[:state]}" +
"#{suburb[:postcode]}") %>
<% end -%>
<% end -%>
The autocompleter view has three (3) hidden
You may also want to cast your eye over the suburb field definition in the new contact view as this is where most of the action is. There are two options to note:
- the :value option which specifies the class name of the element which contains the value to place in attribute (the default would be whatever is within the rendered field which, as we will find out below, will contain more than just the selected suburb)
- the :after_update_element option which specifies a piece of Javascript to execute when the item is selected. You will see that this Javascript executes the Ajax.Autocompleter.extract_value function twice. This function does not exist in Script.aculo.us but is provides an easy way to extract extra values from an autocomplete list.
The Script.aculo.us Autocompleter methods conviently pass the complete contents of the
Additional method for Ajax.Autocompleter class. This can be declared anywhere after the inital Script.aculo.us script inclusion. For my purposes I put this at the top of my application.js file.
Ajax.Autocompleter.extract_value =
function (value, className) {
var result;
var elements =
document.getElementsByClassName(className, value);
if (elements && elements.length == 1) {
result = elements[0].innerHTML.unescapeHTML();
}
return result;
};
So that's all there is to it. If anyone would like me to create a demo of the above code, ask me and, if I get enough requests, I'll put something together.
http://www.edgeblog.net/2006/defending-against-u3-switchblade/
Defending against U3 & Switchblade
Filed under: Security — bill @ 10:22 pm
U3U3 is a fun new technology for USB flash devices. U3 flash drives contain a partition that emulates a CD-ROM drive, where U3 enabled applications are installed. The CD emulation means that these devices will auto-play on most XP, 2000 and 2003 computers, when the drive is inserted. The talented folks over at hak5.org have created several projects, including Switchblade and its younger cousin Hacksaw, which exploit this technology for hacking/pen testing.
U3 reinforces the old security axiom, “if I can touch it, I own it.” Using auto-play with exploit code is nothing new. CDs can be used in this manner. What is new is the ability to run this on a writeable device. As the hak5 guys have proven, this is a deadly combo. Plug your USB drive in, wait for it to suck off password hashes or key files, install a back-door, and be gone. This works even if the screen is locked. One more reason why at some companies, the janitor is the richest guy in the place.
As pen testers, U3 is just one more tool to make our lives easier. As security managers, developing a defense in depth against U3 is difficult. Here are a few suggestions to make it easier. Most of these are just good general security practices, but U3 increases their importance:
1. Assign the least amount of privileges possible to your users. Programs run with U3 execute with the privileges of the logged-on user. Unless, of course, the hacker includes a privilege escalation exploit on the drive.
2. Keep systems patched. This reduces the # of possible exploits.
3. Never leave systems logged in with admin access. Locking the screen does not protect against auto-play. Admins should always log out when done.
4. Disable auto-play. (Instructions below)
5. Restrict USB devices. Several vendors offer solutions to disable USB ports, or restrict them to authorized devices.
* GFI EndPoint Security
* ControlGuard Endpoint Access Manager
* SafeEnd Protector
* Device Lock
* SecureWave Sanctuary
* DeviceWall
* TriGeo USB-Defender
There are mulitple ways to disable auto-run. The best way is to use group policy. Go to computer config>admin templates>system and find the “turn autoplay off option. This option makes a registry entry in HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\Explorer. You can also create this key manually. For stand-alone PCs, the TweakUI PowerToy from Microsoft can also be used. TweakUI offers a disable autoplay option under the “My Computer section.
The last tool in your arsenal is training. Teach your users not bring in USB devices from home, or plug-in flash drives the find or are sent in the mail. This seems like common sense, but several security testers have shown that users will pickup drives on the ground and plug them in to their PCs to see what is on them. The most famous example of this is the test Steve Stasiukonis wrote about in the Dark Reading blog. 20 Flash drives on the ground outside a bank yielded 15 compromised systems. Flash drives work better for this type of test than CDs, because users perceive them as valuable since they are re-writable. A good security education program would prevent this.
If you have other ideas for protecting against flash drives and U3, we’d love to hear about them.
-Bill
Permalink
Thanks for stopping by.
If you found this article useful, please leave a tip.
3 Comments »
1.
Bill said,
October 23, 2006 @ 11:50 am
Nice piece!
2.
edgeblog » 10 New Immutable Laws of IT Security said,
October 23, 2006 @ 4:25 pm
[...] An unsupervised janitor is the richest guy in your company - See rule 4. As I’ve discussed before, a USB key with U3 and a PC with AutoPlay is all it takes to get passwords, install software, and generally 0wn a PC. Couple that with your administrator’s terminals and you have a recipe for disaster. Would you really trust your janitor to do the right thing if I offered him $1,000 to plug a USB drive into a PC for 10 minutes and then bring it back to me? Physical security extends beyond the data center to include every system that has privileged access. How secure are your admin’s home PCs? Your CIO’s? [...]
3.
dante said,
March 4, 2007 @ 6:56 am
hello bill, i had a question about your switchblade article, i am not sure switchblade can run if the screen is password protected. i have tried on two laptops, and it only works when the screen is not locked. if i have missed something, please let me know… thanks for your time.
RSS feed for comments on this post · TrackBack URI
Leave a Comment
Name (required)
E-mail (required)
URI
*
Categories
o Home
o Books
o Compliance
o Data Center Design
o General
o Networks
o Politics
o Popular
o Scripting
o Security
o Software
o Systems
*
Pages
o About
o Donate
*
Archives
o May 2009
o January 2009
o November 2008
o October 2008
o September 2008
o August 2008
o January 2008
o October 2007
o June 2007
o May 2007
o March 2007
o February 2007
o January 2007
o December 2006
o November 2006
o October 2006
o September 2006
*
e-Tip Us!
*
Book Recommendations
o 19 Deadly Sins of Software Security
o Cheat at Windows SysAdmin
o Experts’ Guide to OS/400 & i5/OS Security
o Google Hacking
o Google Maps Applications
o Gray Hat Hacking
o Grid Networks: Advanced Tech
o Hacking iSeries
o How to Break Software Security
o How to Break Web Software
o How to Cheat at Infosec
o Internetworking Technologies Handbook
o Metasploit Toolkit
o Metasploit Toolkit for Penetration Testing
o Protect Your Windows Network
o RFID Security
o Security Warrior
o SELinux By Example
o Silence on the Wire
o Stealing the Network: How to Own a Continent
o Stealing the Network: How to Own a Shadow
o Stealing the Network: How to Own an Identity
o Stealing the Network: How to Own the Box
o The Security Development Lifecycle
o Ubuntu Hacks
o Windows Powershell in Action
o WordPress 2 Quickstart
o Writing Secure Code, 2nd ed.
o Zen of CSS Design
*
Friends
o EdgeBack
o edgeproxy
o Gadget Workshop
o iQuotient
o IronScale
o R3publicans
o Raging Wire
o Secure Insanity
o The Digerati Life
*
Security Links
o CERT/CC
o CVE
o Foundstone
o Full Disclosure
o Infosec Institute Blog
o OSSTMM
o Switchblade
*
CERT⁄CC
o Microsoft Releases Service Pack 2 for Windows Vista and Windows Server 2008
o Novell Releases Updates for GroupWise
o NSD DNS Buffer Overflow Vulnerability
o Cisco Releases Security Advisory for CiscoWorks TFTP Vulnerability
o Mac OS X Includes Known Vulnerable Version of Java
*
Northern Cal Jobs (Dice)
o SW Engr 2
o SW Engr 2
o Technical Support
o Entrepreneurial Leader
o Oracle DBA
o Installation Technician
o Oracle DBA
o Software Engineer
o Systems Engineer Intern/Graduate - College
o CIO
Xobni outlook add-in for your inbox
Digg!
©2006 William L. Dougherty • Design based on Corporate Pro by Mystical Twilight ·
--
map{ map{tr|10|# |;print} split//,sprintf"%.8b\n",$_}
unpack'C*',unpack'u*',"5`#8<3'X`'#8^-@`<-CPP`#8V/C8`"
Scriptaculous Autocomplete Page Jump Using Arrow Keys
Tags: CSS, JavaScript, Scriptaculous
When you use overflow:auto in your css in conjunction with Scriptaculous’ Autocomplete, there is a bug that makes the entire page jump around when you use the arrow keys on your keyboard to navigate up and down through the suggestion list. This bug normally appears only when the page itself is long enough to require a scroll bar.
I managed to come up with a very clean working solution by hacking the controls.js file that comes with scriptaculous. The solutions requires replacing the markPrevious and markNext functions and adding a small line of code to the updateChoices function.
Currently, markPrevious and markNext are telling the page to jump around like that, and I’m not sure why! As far as I can tell (please let me know otherwise) this solution could be included in the scriptaculous without breaking a thing (hint, hint to the scriptaculous team).
To implement the solution, replace:
markPrevious: function() {
if(this.index > 0) this.index--;
else this.index = this.entryCount-1;
this.getEntry(this.index).scrollIntoView(true);
},
markNext: function() {
if(this.index < this.entryCount-1) this.index++;
else this.index = 0;
this.getEntry(this.index).scrollIntoView(false);
},
With:
markPrevious: function() {
if(this.index > 0) {this.index--;}
else {
this.index = this.entryCount-1;
this.update.scrollTop = this.update.scrollHeight;
}
selection = this.getEntry(this.index);
selection_top = selection.offsetTop;
if(selection_top < this.update.scrollTop){
this.update.scrollTop = this.update.scrollTop-selection.offsetHeight;
}
},
markNext: function() {
if(this.index < this.entryCount-1) {this.index++;}
else {
this.index = 0;
this.update.scrollTop = 0;
}
selection = this.getEntry(this.index);
selection_bottom = selection.offsetTop+selection.offsetHeight;
if(selection_bottom > this.update.scrollTop+this.update.offsetHeight){
this.update.scrollTop = this.update.scrollTop+selection.offsetHeight;
}
},
Now find the updateChoices function and just after the code this.stopIndicator(); add this.update.scrollTop = 0; so that it looks like this:
this.stopIndicator();
this.update.scrollTop = 0;
this.index = 0;
I’ve tested in FF 2.0.0.4, FF 3.0.5, Chrome 1.0.154.43, Safari 3.2.1, Opera 9.5.1, IE 7.0.5730.13 and IE 6.0.2600 without problems.
Bookmark and Share
--
map{ map{tr|10|# |;print} split//,sprintf"%.8b\n",$_}
unpack'C*',unpack'u*',"5`#8<3'X`'#8^-@`<-CPP`#8V/C8`"
Subscribe
Posts
All Comments
share
