Slow compound mysql update query -

-

we're doing update query between 2 database tables , ridiculously slow. in: take 30 days perform query.

one table, lab.list, contains 940,000 records, other, mind.list 3,700,000 (3.7 million) update sets field when 2 between conditions met. query: 

update lab.list l , mind.list m set l.locid = m.locid  l.longip between m.startipnum , m.endipnum , l.date between "20100301" , "20100401" , l.locid = 0

as now, query performing 1 update every 8 seconds.

we tried mind.list table in same database, doesn't matter query time.

update lab.list l, lab.mind m  set l.locid = m.locid   longip between m.startipnum , m.endipnum , date between "20100301" , "20100401" , l.locid = 0;

is there way speed query? imho should make 2 subsets of databases: mind.list.longip between m.startipnum , m.endipnum lab.list.date between "20100301" , "20100401"

and update values these subsets. somewhere along line think made mistake, where? maybe there faster query possible?

we tried log_slow_queries, shows indeed examining 100s of millions of rows, going way 3331 gigarows.

tech info:

  • server version: 5.5.22-0ubuntu1-log (ubuntu)
  • lab.list has indexes on locid, longip, date
  • lab.mind has indexes on locid, startipnum , m.endipnum
  • hardware: 2x xeon 3.4 ghz, 4gb ram, 128 gb ssd (so should not problem!)

i first of try index mind on startipnum, endipnum, locid in order. locid not used in selecting mind, if used update.

for same reason i'd index lab on locid, date , longip (which isn't used in first chunking, should run on date) order.

then kind of datatype assigned startipnum , endipnum? ipv4, it's best convert integer , use inet_aton , inet_ntoa user i/o. assume did this.

to run update, might try segment m database using temporary tables. is:

* select records of lab in given range of dates locid = 0 temporary table table1. * run analysis on table1 grouping ip addresses first n bits (using , suitable mask: 0x80000000, 0xc0000000, ... 0xf8000000... , on, until find  have divided "suitable" number of ip "families". these will, , large, match startipnum (but that's not strictly necessary). * have divided in 1000 families of ip. * each family: *    select ips table1 table3. *    select ips matching family mind table2. *    run update of matching records between table3 , table2. should take place in 1 hundred thousandth of time of big query. *    copy-update table3 lab, discard table3 , table2. * repeat next "family".

it not ideal, if improved indexing not help, don't see many options.


Comments

Post a Comment

Popular posts from this blog

c# - SVN Error : "svnadmin: E205000: Too many arguments" -

-
i trying reposotries using c# code process svncommand = null; var psi = new processstartinfo("svnadmin"); psi.redirectstandardoutput = true; psi.redirectstandarderror = true; psi.useshellexecute = false; psi.arguments= @"dump c:\repositories\myrepo > c:\temp\myrepodumpfile.dump"; using (svncommand = process.start(psi)) { var myoutput = svncommand.standardoutput; var myerror = svncommand.standarderror; debug.write("output :" + environment.newline + environment.newline + myoutput.readtoend() + environment.newline + "error :" + environment.newline + environment.newline + myerror.readtoend()); svncommand.close(); } when use dump command commandline svnadmin dump c:\repositories\myrepo > c:\temp\myrepodumpfile.dump it works fine when try ...
Read more

c# - Copy ObservableCollection to another ObservableCollection -

-
how copy observablecollection item observablecollection without reference of first collection? here observablecollection item value changes affecting both collections. code private observablecollection<ratemodel> _allmetalrate = new observablecollection<ratemodel>(); private observablecollection<ratemodel> _metalrateondate = new observablecollection<ratemodel>(); public observablecollection<ratemodel> allmetalrate { { return this._allmetalrate; } set { this._allmetalrate = value; notifypropertychanged("metalrate"); } } public observablecollection<ratemodel> metalrateondate { { return this._metalrateondate; } set { this._metalrateondate = value; notifypropertychanged("metalrateondate"); } } foreach (var item in metalrateondate) allmetalrate.add(item); what causing , how can solve it? you need clone object referenced item before it's a...
Read more

All overlapping substrings matching a java regex -

-
is there api method returns (possibly overlapping) substrings match regular expression? for example, have text string: string t = 04/31 412-555-1235; , , have pattern: pattern p = new pattern("\\d\\d+"); matches strings of 2 or more characters. the matches are: 04, 31, 412, 555, 1235. how overlapping matches? i want code return: 04, 31, 41, 412, 12, 55, 555, 55, 12, 123, 1235, 23, 235, 35. theoretically should possible -- there obvious o(n^2) algorithm enumerates , checks substrings against pattern. edit rather enumerating substrings, safer use region(int start, int end) method in matcher . checking pattern against separate, extracted substring might change result of match (e.g. if there non-capturing group or word boundary check @ start/end of pattern). edit 2 actually, it's unclear whether region() expect zero-width matches. specification vague, , experiments yield disappointing results. for example: string line = "xx90xx"; strin...
Read more