#maria Log v0.1

logs Catalogue

This page loads from the db, refresh as and when. It displays 24 hrs
For older data http://marialog.archivist.info/previous.php, logs go back to 2009-01-18
Some user stats
Table logs
date_time
user
message
2015-05-28 03:54:42
jamescarr
hey quick question... we have servers running 92 processes, each process maintains a mysql connection...
2015-05-28 03:54:55
jamescarr
would maxscale help in providing a local proxy per box to pool the connections?
2015-05-28 03:55:11
jamescarr
so that each machine only has, say, 10 connections to the database vs. 92
2015-05-28 04:01:04
danblack
the thing about connections is they have state (wheither its used or not).
2015-05-28 04:01:40
danblack
jamescarr: what problem are you trying to solve? Was each process maintaining a connection attempting to solve a problem?
2015-05-28 04:01:58
jamescarr
they're queue workers...
2015-05-28 04:02:12
jamescarr
the problem I'm trying to solve is that during high load we have too many connections
2015-05-28 04:02:46
jamescarr
by high load I mean our autoscale group scales up and then stat getting some refused connections against RDS
2015-05-28 04:02:55
jamescarr
that's the specific problem we're hoping to solve
2015-05-28 04:04:37
danblack
so the 92 processes aren't all the connections used right. Have you optimised the queries as much as possible?
2015-05-28 04:04:51
danblack
are you using read slaves?
2015-05-28 04:05:39
danblack
i'm seeing connections as a symptom of the problem but not the problem itself.
2015-05-28 04:06:00
jamescarr
we are using read slaves
2015-05-28 07:37:02
rrva
With semanage permissive -a mysqld_safe_t and semanage permissive -a mysqld I should be covered for MariaDB/Galera on RHEL6 right? /etc/init.d/mysqld has a different context, it's in initrc_exec_t...
2015-05-28 07:44:55
danblack
rrva: sounds about right. I might need to update https://mariadb.com/kb/en/mariadb/what-to-do-if-mariadb-doesnt-start/ to cover mysqld_safe_t
2015-05-28 07:46:52
danblack
actually mysqld_safe_t isn't actually galera specific afaik so may not be needed.
2015-05-28 07:47:02
danblack
feedback based on exprience welcome
2015-05-28 07:54:53
rrva
danblack: it is needed, have logs to prove it. /etc/my.cnf cannot be read by /usr/bin/my_print_defaults otherwise
2015-05-28 07:55:11
danblack
ack. thanks
2015-05-28 07:58:56
rrva
danblack: wanting to try to fix https://mariadb.atlassian.net/browse/MDEV-7631 myself.. either by workaround or doing some patches in sql/wsrep_utils.cc. Comments welcome! Have been speculating if using --init-file=change_password.sql makes it happen.
2015-05-28 08:00:02
rrva
danblack: about the my_print_defaults, as you can imagine causes weird fallbacks to default config values
2015-05-28 08:00:03
danblack
not sure. about to run off for dinner.
2015-05-28 08:32:33
LondonAppDev
Has anyone managed to get MariaDB 10 to work with MySQL-Python ?
2015-05-28 08:34:53
serg
LondonAppDev: what error are you getting?
2015-05-28 08:35:38
LondonAppDev
One sec, let me copy it to gist
2015-05-28 08:35:43
LondonAppDev
Just need to re-produce it.
2015-05-28 08:35:54
LondonAppDev
Sorry should have been ready...
2015-05-28 08:37:23
LondonAppDev
serg: I get this error when I run `pip install mysql-python`
2015-05-28 08:37:24
LondonAppDev
https://gist.github.com/LondonAppDev/fc64bbf28199b3d22358
2015-05-28 08:37:36
LondonAppDev
I am using Debian 8
2015-05-28 08:42:48
serg
it's a bug, in mariadb and in mysql (we've inherited it, apparently). I'll report it and we'll fix it, but meanwhile as a workaround you can add 1 to that line on /usr/include/mysql/my_config.h:438
2015-05-28 08:42:58
serg
and other similar lines, if needed
2015-05-28 08:51:26
LondonAppDev
serg: Oh, thanks a ton :)
2015-05-28 08:51:38
LondonAppDev
Are you a contributor on MariaDB?
2015-05-28 08:51:48
serg
a core developer
2015-05-28 08:52:39
LondonAppDev
Thank you for the fantastic database option!
2015-05-28 08:55:07
jkavalik
Is there some reason why count group by expression uses temp table where the same with "with rollup" does not? (and is slower without it) http://pastebin.com/sXPpTbun
2015-05-28 08:55:42
jkavalik
(* slower without tmp table)
2015-05-28 09:35:17
vondel
is there a way to reset the status-count 'Innodb_deadlocks' in maria 5.5.34? "flush status" didn't reset it
2015-05-28 09:44:44
danblack
vondel: if flush status didnt' reset it it probably can be done by server restart only.
2015-05-28 09:52:16
vondel
pity
2015-05-28 10:47:28
danblack
vondel: i did see it was declared as SHOW_LONG in the 10.0 source code so it might be flushable there.
2015-05-28 10:56:28
vondel
ok
2015-05-28 10:56:37
vondel
another reason to upgrade :)
2015-05-28 10:56:49
vondel
not sure when it will happen though :(
2015-05-28 10:57:33
tanj
well, running 5.5.34 is probably already a big problem in itself
2015-05-28 10:58:41
vondel
long history of "if it works, don't fix it" in our company :)
2015-05-28 10:59:09
vondel
so it's good to build a list of things that don't "work" in the antique versions :)
2015-05-28 11:11:29
danblack
so you haven't hit the memory leak i'm guessing. sigh. the hours spent to narrowing it down. :-)
2015-05-28 12:13:09
danblack
i'm reposition a 10.0 gtid slave. I've done a reset slave; SET GLOBAL gtid_slave_pos = ...;change master to master_use_gtid=slave_pos;start slave io_thread; i'm a bit nervious about starting a sql thread as the show processlist is showing the old sql work like | 5404 | system user | | NULL | Connect | 102509 | Waiting for work from SQL thread | UPDATE event INNER
2015-05-28 12:13:33
danblack
are those sql thread processes going to die/rollback on a start slave sql_thread?
2015-05-28 12:29:51
danblack
well. i tried. i think they died.
2015-05-28 12:31:25
tanj
danblack: ah, oh that's some kind of bug I had as well.
2015-05-28 12:31:59
tanj
danblack: replication threads don't clear their status correctly, for some reason.
2015-05-28 12:44:24
jaijop
hello
2015-05-28 12:44:35
jaijop
Which has more performance.....
2015-05-28 12:44:37
jaijop
?
2015-05-28 12:45:06
jaijop
A multi-master mariadb cluster on Vmware VM or Physical vm?
2015-05-28 12:46:40
tanj
jaijop: depends on several factors. VM is not one of them
2015-05-28 12:46:43
danblack
tanj: did you notice it was status only or was there some reality behind those indicators?
2015-05-28 12:47:25
tanj
danblack: it was just the last statement executed, just that it didn't clear from processlist
2015-05-28 12:47:50
tanj
kolbe has pointed me to some bug upstream, although i'm not sure which one it was...
2015-05-28 12:48:04
danblack
ok. all good :-) thanks
2015-05-28 12:48:11
jaijop
Actually... I have configured mariadb master-master cluster with 3 nodes on both vmware instance and on Physical Machines
2015-05-28 12:48:21
tanj
i don't know if it's referenced in MDEV. elenst? any chance you might know about that
2015-05-28 12:48:42
jaijop
But when i do a load test with 800vu....Physical machines have better performance than Vm instances
2015-05-28 12:49:08
tanj
jaijop: do the VM have the same specs as the physical machines?
2015-05-28 12:49:14
jaijop
yes
2015-05-28 12:49:17
jaijop
more than that
2015-05-28 12:49:26
jaijop
i mean on PM we have 8gb ram
2015-05-28 12:49:35
jaijop
but on VM ware 16 gb and 4 core
2015-05-28 12:49:55
tanj
how about the disks?
2015-05-28 12:49:56
jaijop
but physical machines show better performance
2015-05-28 12:50:05
tanj
try to compare what's comparable
2015-05-28 12:50:11
jaijop
disk is more in vm....200GB
2015-05-28 12:50:15
tanj
not the size
2015-05-28 12:50:17
tanj
the type of disk
2015-05-28 12:50:19
tanj
speed etc
2015-05-28 12:50:27
archivist_herron
is the disk shared
2015-05-28 12:50:33
jaijop
yes
2015-05-28 12:50:35
jaijop
on vmware
2015-05-28 12:50:38
jaijop
not in pm
2015-05-28 12:51:31
tanj
you can have almost similar performance on VM if you use dedicated resources instead of shared resources
2015-05-28 12:51:42
tanj
in case of shared resources I would expect the physical host to be faster, ofc
2015-05-28 12:52:05
danblack
couldn't find a mdev.
2015-05-28 12:52:43
tanj
danblack: I'll check with elenst or serg if they come around. I suppose you're getting in bed soon :)
2015-05-28 12:52:59
danblack
it would be sensible :-)
2015-05-28 12:55:09
danblack
i'm just glad it was harmless. theres many more bugs i care about that are dawdling (MDEV-7786/MDEV-6735 cough).
2015-05-28 12:56:09
elenst
tanj: check what? where should i start reading from?
2015-05-28 12:56:13
elenst
oh wait i see above
2015-05-28 12:57:00
tanj
elenst: it's mostly cosmetic, for some reason in system threads, the last executed statement doesn't clear.
2015-05-28 12:57:42
tanj
i've seen that with galera replication threads mostly, but it seems it happens with regular slave sql threads as well...
2015-05-28 12:57:56
elenst
nope i don't see where it starts. tanj: when did you start describing the issue?
2015-05-28 12:58:23
danblack
it was me. MDEV-6327 is the closest i've found
2015-05-28 12:58:34
danblack
21:13 < danblack> i'm reposition a 10.0 gtid slave. I've done a reset slave; SET GLOBAL gtid_slave_pos = ...;change master to master_use_gtid=slave_pos;start slave io_thread; i'm a bit nervious about starting a sql thread as the show processlist is showing the old sql work like | 5404 | system user | | NULL | Connect | 102509 | Waiting for work from SQL thread | UPDATE event INNER
2015-05-28 12:58:37
elenst
ah
2015-05-28 12:59:43
elenst
weird
2015-05-28 13:00:00
tanj
elenst: i also had a Remote DBA customer reporting that, although i must dig in the issues to find what it was...
2015-05-28 13:01:04
danblack
i did have a replication error before i did the reset. that's probably the point at which is should of cleared - unless it was leaving thos parallel threads in a state waiting for a fix.
2015-05-28 13:01:04
jkavalik
we have most galera threads in "closing tables" state after they process first replicated "item", with time growing until they process next time - thats it?
2015-05-28 13:01:05
tanj
| 1 | system user | | NULL | Sleep | 73 | closing tables | insert into t1 values(1) | 0.000 |
2015-05-28 13:01:21
tanj
jkavalik: yeah, it sounds like a similar issue.
2015-05-28 13:01:37
elenst
so, the slave was stopped and reset, but these threads stayed in the processlist?
2015-05-28 13:01:45
danblack
yes
2015-05-28 13:01:49
elenst
hm
2015-05-28 13:01:56
elenst
maybe knielsen knows anything about it
2015-05-28 13:02:35
jkavalik
tanj, yep, seems same, our time gets usually max to 4 on 10 threads, but all the time "closing tables" unless processlist shows some processing at all (hard to get, they are fast)
2015-05-28 13:02:36
elenst
danblack: and yeah, sorry for MDEV-6327, i guess it got burried under major issues
2015-05-28 13:02:43
elenst
if you want, i can increase the priority
2015-05-28 13:02:53
elenst
can't promise it will help, though
2015-05-28 13:04:40
danblack
yeh - there's only so much you can increase priority on thing with limited resources to solve them.
2015-05-28 13:07:39
danblack
elenst: got an easier question for you if you have time. i'd like a mtr test to do a select statement without its result ending up in the .result. Can't use into outfile as its a QC patch. suggestions?
2015-05-28 13:08:06
elenst
--disable_result_log
2015-05-28 13:08:46
danblack
and i assume --enable_result_log at the end. great. was fairly sure it would be easy.
2015-05-28 13:08:48
elenst
before the query
2015-05-28 13:08:50
elenst
right
2015-05-28 13:08:54
elenst
yeah that was easy
2015-05-28 13:09:03
elenst
elenst likes easy
2015-05-28 13:11:45
danblack
well thanks. would of taken me quite a while to find.
2015-05-28 13:12:11
danblack
danblack wonders how devs aren't tripping over MDEV-4447 every time they switch branches now.
2015-05-28 13:49:49
amcorreia
hi, any way to use just 2 servers in cluster?
2015-05-28 13:50:30
danblack
not sanely. look up split brain.
2015-05-28 13:51:58
danblack
of course what defination of cluster you use will determine what you are after a lot more.
2015-05-28 13:54:18
tanj
amcorreia: if it's about mariadb galera cluster you can have 2 physical servers + 1 arbitrator (still requires a third server though, but not as much resources...)
2015-05-28 13:55:57
roidelapluie
Hello
2015-05-28 13:56:12
amcorreia
I can choose who is the arbitrator?
2015-05-28 13:56:28
amcorreia
(to get the server with less resources)
2015-05-28 13:56:46
roidelapluie
How to explain Opened table raising while Open table is only 10% of the Open table cache?
2015-05-28 13:57:17
roidelapluie
How to explain "Opened table" raising while "Open table" is only 10% of the "Open table cache"?
2015-05-28 14:00:01
danblack
roidelapluie: are you hitting a file-limit or innodb_open_files?
2015-05-28 14:01:15
jkavalik
are temporary tables counted to "opened" ?
2015-05-28 14:02:15
roidelapluie
danblack: innodb_open_files is the same value as the table_open_cache
2015-05-28 14:08:14
roidelapluie
another question
2015-05-28 14:08:37
roidelapluie
why do I soo performance schema in open tables while it is disabled
2015-05-28 14:11:53
danblack
query cache miss. answer not found.
2015-05-28 14:30:24
firc12a4eb60
Does anyone know why I'd get an error 1236 on my slave after I move my binlogs?
2015-05-28 14:31:17
danblack
because your slave hadn't finished with them
2015-05-28 14:32:26
firc12a4eb60
danblack: Fair enough but if I move the files to a different directory and change the path to my binlogs when I start up the master again it should work right?
2015-05-28 14:33:33
knielsen
firc12a4eb60: the slave keeps the name of the binlog file (I think it's full path?), seen in SHOW SLAVE STATUS, and in the error log when connecting
2015-05-28 14:33:37
danblack
the .index file sometimes contains a full path.
2015-05-28 14:33:44
knielsen
right, that too
2015-05-28 14:34:51
danblack
i don't think the slave stores a full path.
2015-05-28 14:35:58
knielsen
you're probably right. In any case, using GTID the path is not used at all ;-)
2015-05-28 14:36:20
knielsen
(but it still needs to be correct in .index on the master)
2015-05-28 14:36:31
amcorreia
150528 9:36:02 [Warning] WSREP: no nodes coming from prim view, prim not possible
2015-05-28 14:36:49
amcorreia
why this error? In bootstrap process
2015-05-28 14:37:04
amcorreia
I have 3 nodes now
2015-05-28 14:40:18
danblack
yeh, gtid much nicer. big deployment of it next week. anyway. gn.
2015-05-28 14:41:33
mdhas
Hmm so how does one move binlogs from one disk to another and not end up with the slave crying foul?
2015-05-28 14:42:37
danblack
mdhas: if you can symlink old location to new or mount --bind. alternately edit index files on shutdown master.
2015-05-28 14:43:57
mdhas
danblack: The index file is editable?
2015-05-28 15:51:18
syranide
anyone know when the debian jessie repository for mariadb is going up?
2015-05-28 16:08:05
siriusboehm
hey, one guy did a stupid mistake and upgraded mariadb-galera-server on one node accidently, which resulted in a complete cluster stucking, so we did upgraded all nodes, resynced somehow and it worked. Now after a few days we are still getting problems(e.g. complete deadlock), it is possible that the fail-upgrade destroyed something?
2015-05-28 16:55:19
tmkt
how does mysqlbinlogmove differ from cp -R relay_logs/* newdir/relay_logs/; chown mysql:mysql newdir/relay_logs/* ?
2015-05-28 16:55:36
tmkt
Last week i moved relay logs from one location to antoher via the cp route...without issue
2015-05-28 16:55:53
tmkt
but had to move them again recently and whenever i start mysql back up its slaves can't find the relay logs
2015-05-28 16:59:26
kolbe
rrva: are you around?
2015-05-28 17:00:00
kolbe
rrva: i'm wondering how/why your /etc/my.cnf file has unconfined_u:object_r:admin_home_t:s0 ... on CentOS 6.6 mine is system_u:object_r:mysqld_etc_t:s0
2015-05-28 17:19:05
tmkt
nm..i seem to have forgot to update the index file
2015-05-28 17:32:33
canned
I'm trying to put together a summary of mariadb vs mysql differences - I've seen the usual comparison charts, but I'm just wondering, are there any internal differences with the way that buffering/caching is handled or stored in memory and on disk?
2015-05-28 17:34:27
canned
Or is that essentially just going to depend on the engine and nothing else?
2015-05-28 17:36:18
grknight
canned: one thing is, by default, temporary tables on disk differ in type. mariadb uses aria and oracle uses myisam last i heard
2015-05-28 17:37:08
canned
ahhh that's right :) I seem to remember reading about that. I think MySQL 5.6+ uses InnoDB for the tmp tables now, but prior to that it was MyISAM, yep
2015-05-28 17:39:25
x86iac
canned: i'd say the difference is 3 echo mariadbs | wc -c ; echo mysql | wc -c
2015-05-28 17:39:44
canned
badum pshh
2015-05-28 17:40:05
x86iac
i use myisam with mariadb - easier all the way around for dev
2015-05-28 17:40:38
canned
mostly trying to get an idea of what performance differences a lazy client might have, eg. the ones that upgrade from MySQL, have unoptimized databases with MyISAM and InnoDB tables pre-existing, and don't want to do any tuning or my.cnf work
2015-05-28 17:42:38
canned
they know mariaDB is there, and they know it's where we're headed, so they're going to gravitate towards it, and that's great, I just want to be able to give them an honest explanation of what they might expect
2015-05-28 17:43:30
canned
or I guess a "realist" explanation would be a better word.. knowing that the least-effort scenario is going to be the one that comes up the most
2015-05-28 17:54:21
x86iac
x86iac for one believes canned
2015-05-28 18:01:14
amcorreia
In cluster, how badly a low resource server (or in a different datacenter) can impact in cluster performance?
2015-05-28 18:03:59
canned
x86iac: haha.. is there an ulterior motive that I should be aware of?
2015-05-28 18:08:35
x86iac
canned: i for used 'the least-effort scenario' when i moved from mysql to mariadb and never had a real problem - just a comment out of something in the my.cnf and things went smoothly
2015-05-28 18:16:53
canned
yeah I don't expect any problems with the upgrade really. its about as drop-in as it gets, I just don't want to beef up their expectations too high with a bunch of optimization talk if a lot of it won't apply to their environments, mostly.
2015-05-28 18:18:05
canned
so I was trying to get an idea of what they might see differently right out of the box
2015-05-28 18:18:25
canned
so to speak
2015-05-28 18:20:59
canned
and a lot of the big improvements in mariadb looked like they required the user to deliberately and consciously take advantage of them, either via simply enabling it, or changing the way that they structure queries (or just building it with that in mind from the beginning)
2015-05-28 18:23:07
x86iac
canned: ? you tell them it is a very tricky procedure that requires utter solitude - ie don't disturb me while i do it and i need lots of pizza and soda - then close the door behind them and have fun .. in an hour come out all sweaty and perplexed and wipe you hands on a towel, throw the towel down and say in a loud voice - it is done .. finally .. and pay me the $5000 and ..
2015-05-28 18:25:17
canned
hahaha.. brilliant. I've been doing it wrong for a long time apparently
2015-05-28 18:30:52
x86iac_
yea don't be a troublemaker - get with the program ;)
2015-05-28 18:31:34
x86iac_
oh and get the laugh and head shake down when they suggest mssql
2015-05-28 18:32:11
x86iac_
the more you can shame mssql people the more you can charge to upgrade them to mariadb
2015-05-28 18:33:44
canned
"more like, M$$QL, am I right"
2015-05-28 18:35:10
canned
"that'll be $7500 for the mariadb upgrade.. cash only please"
2015-05-28 18:36:04
canned
...plus the $200 convenience and transportation fees.
2015-05-28 18:56:40
x86iac_
the worst people are the oracle/apex crowd - don't bother with them
2015-05-28 20:02:42
jafa
hi guys
2015-05-28 20:03:16
jafa
i am seeing a problem where mariadb starts fine, but takes longer than ubuntu "service mysql start" allows
2015-05-28 20:03:33
jafa
normally this isn't a problem - I ignore the error and the db starts fine
2015-05-28 20:04:39
jafa
problem is an update was released and apt-get upgrade keeps failing and not finishing the mariadb/galeria update because it thinks the service start is failing
2015-05-28 20:11:07
bytee
jafa: do you know why its taking so long, like what is the reason for the stall ?
2015-05-28 20:14:04
jafa
initial galeria sync I think
2015-05-28 20:14:40
jafa
can take several minutes before the db is up
2015-05-28 20:16:17
jafa
I got the db back up, but if anyone runs an apt-get command it will kill it again
2015-05-28 20:16:43
jafa
because apt-get thinks it has an incomplete package install it needs to finish
2015-05-28 20:21:12
bytee
svoj: i'm in touch with Delveri. all is well
2015-05-28 20:21:29
svoj
bytee: great, thanks a lot!
2015-05-28 20:21:39
bytee
jafa: hmm, this is clearly an issue. i wonder if this is repeatable with percona xtradb cluster, or if there is something we can do to avoid this problem
2015-05-28 20:21:55
bytee
jafa: is there a bug report against the package in ubuntu?
2015-05-28 20:22:59
bytee
montywi: ping
2015-05-28 20:35:35
mgriffin
jafa: just curious about a few things as i have not encountered this: do you stop the service before starting apt? is the node doing IST or SST during startup? do you disable the debian-start script (if this runs by default on mariadb-galera)?
2015-05-28 22:23:54
BlaDe^
kolbe: https://mariadb.atlassian.net/browse/MDEV-8236 -- just FYI
2015-05-28 22:26:09
kolbe
BladedThesis: thanks!
2015-05-28 22:45:21
jafa
mgriffin: stop before apt - tried both way, doesn't make a difference as apt stops the service and stopping works fine
2015-05-28 22:46:35
jafa
ist / sst?
2015-05-28 22:48:38
jafa
is there a better approach to running a db for a webserver?
2015-05-28 22:49:03
mgriffin
jafa: i wasn't sure if apt would "start" an already stopped service, is why i asked. ist should be fast compared to sst.
2015-05-28 22:49:10
jafa
currently running two mariadb+galeria servers and garb on a thrid machine
2015-05-28 22:49:35
jafa
it starts even if already stopped
2015-05-28 22:49:58
jafa
then the script fails because "service mysql start" reported failure
2015-05-28 22:50:34
jafa
having a list of problems, this one being the one that took our website down today
2015-05-28 22:50:59
jafa
list of problems = list of mariadb+galera problems
2015-05-28 22:51:08
jafa
everything else is solid
2015-05-28 22:52:51
jafa
the other big one is I have to shut down mariadb on one server so it only has one node running when I do an input (mysqldump generated sql)
2015-05-28 22:53:02
jafa
otherwise it locks up
2015-05-28 22:53:23
jafa
not every time, but a good 75% of the time, and in different places each time
2015-05-28 22:53:43
jafa
and that lockup takes out all databases, not just the one being imported
2015-05-28 22:54:54
jafa
been using mysql for years without issue (single server)
2015-05-28 22:55:17
mgriffin
how are you using the two nodes with your app?
2015-05-28 22:55:20
tanj
well using galera is obviously trickier, especially when people configure it by themselves
2015-05-28 22:55:31
tanj
there are lots of mistakes being done
2015-05-28 22:56:43
jafa
mgriffin: mix of php and c++ code
2015-05-28 22:57:21
jafa
and the normal mysql shell for importing mysqldumps
2015-05-28 22:57:56
mgriffin
jafa: i mean does the application connect to both nodes? what do they do after connecting (reads, writes)
2015-05-28 22:58:26
jafa
both nodes are used for reads and writes
2015-05-28 22:59:02
tanj
first mistake
2015-05-28 22:59:16
tanj
don't use both nodes for writes, unless you know what you are doing
2015-05-28 23:00:13
jafa
goal is two symertrical machines where either can be taken off line at any time
2015-05-28 23:00:44
tanj
that's ok
2015-05-28 23:00:54
jafa
for example - user requests a forum url, nginx strip https, then hits varnish. If both master servers are online varnish will round-robin the requests to both servers
2015-05-28 23:01:04
tanj
directing writes to only one machine doesn't contradict this principle
2015-05-28 23:01:38
tanj
varnish routes http requests, so i don't see the correlation with the database
2015-05-28 23:01:44
jafa
the two backend servers are used for both apache and db
2015-05-28 23:01:54
tanj
bad idea #2
2015-05-28 23:02:53
jafa
the alternative is to set up VMs to provide fake seperation
2015-05-28 23:03:17
jafa
apache on master1 only talks to db on master1
2015-05-28 23:03:29
tanj
I wouldn't call that fake, because there is resource isolation
2015-05-28 23:03:42
tanj
at least your db cannot use webserver resources and vice versa
2015-05-28 23:04:01
jafa
what resources do you have in mind?
2015-05-28 23:04:09
jafa
ram and disk isn't an issue
2015-05-28 23:04:14
jafa
cpu will be shared anyway
2015-05-28 23:04:50
jafa
machines are significantly overpowered because the used to work without the varnish frontend nodes
2015-05-28 23:05:48
tanj
cpu / ram are not shared in the sense you believe they are
2015-05-28 23:06:44
tanj
anyway we are digressing, 1st probable cause is the fact that you are directing writes to both masters
2015-05-28 23:06:47
jafa
load average is currently 0.08 on a 16-core machine
2015-05-28 23:08:00
jafa
using 14GB RAM + 5GB cache out of 48GB physical
2015-05-28 23:08:08
tanj
i don't feel the need to prove the architecture point further. there's a good reason why architecture designs involves service separation
2015-05-28 23:08:45
jafa
galera doesn't support writes to two nodes?
2015-05-28 23:09:15
tanj
galera uses a certification mechanism, which means that writing to both nodes at the same times may create conflicts
2015-05-28 23:10:33
tanj
just go to both of your servers and check those status variables: "wsrep_local_bf_aborts" and "wsrep_local_cert_failures"
2015-05-28 23:10:57
tanj
if they are > 0 then you're having issues
2015-05-28 23:12:29
jafa
aborts = 3
2015-05-28 23:12:45
jafa
cert failures = 17
2015-05-28 23:13:21
tanj
that's not much, so writing on both nodes may actually work for you
2015-05-28 23:13:42
jafa
how do you get an app to write to one node if it is working, and another node if the primary isn't?
2015-05-28 23:14:08
tanj
use a LB?
2015-05-28 23:14:18
jafa
mysql LB?
2015-05-28 23:14:26
tanj
yeah MaxScale for instance
2015-05-28 23:14:41
tanj
you can even split your reads and writes
2015-05-28 23:15:12
tanj
iirc your issue is with loading dumps?
2015-05-28 23:15:48
jafa
yes, that causes the db to lock up
2015-05-28 23:16:04
tanj
there might be something which locks stuff, although without looking at process lists and error logs that's a long shot
2015-05-28 23:16:21
tanj
i haven't seen lockups in a while
2015-05-28 23:16:27
tanj
do you run a recent version of Galera?
2015-05-28 23:16:33
jafa
if I remember correctly the logs showed a lock held
2015-05-28 23:17:50
tanj
that could be a conflict, although that should resolve by itself
2015-05-28 23:18:01
tanj
unless you're doing big DDLs or things like that
2015-05-28 23:18:09
tanj
or you run a super old and buggy version of Galera
2015-05-28 23:18:16
jafa
also the import is to a database that is only ever read
2015-05-28 23:18:41
tanj
if you ever find out the error messages just lmk. i gotta go for now
2015-05-28 23:19:48
jafa
is it normal for mariadb to take a few mins to start (ie problem is that service start timeout is too short), or should I be looking at mariadb?
2015-05-28 23:20:53
x86iac_
minutes? no not normal - what else you have running on the system? and what is your pre-empt in the kernel?
2015-05-28 23:21:09
jafa
server is pretty much idle
2015-05-28 23:21:25
jafa
the db starts, but takes a few mins if it isn't the first node
2015-05-28 23:22:03
x86iac_
anything in the log?
2015-05-28 23:22:19
jafa
what should I be looking for?
2015-05-28 23:23:36
jafa
I will force a restart now...
2015-05-28 23:23:43
x86iac_
does starting it at another time cause same delay? what about starting httpd or another server
2015-05-28 23:24:05
jafa
it always takes a while
2015-05-28 23:24:12
jafa
other services are finwe
2015-05-28 23:24:47
jafa
starting again
2015-05-28 23:24:58
jafa
huh - came up quickly
2015-05-28 23:25:18
jafa
trying apt-get upgrade again
2015-05-28 23:25:38
jafa
normally it takes a few mins
2015-05-28 23:26:14
jafa
trying the other server
2015-05-28 23:27:51
jafa
weird - that worked... started back up fast enough for apt-get to finish
2015-05-28 23:29:49
x86iac_
any dns problem in this?
2015-05-28 23:30:39
jafa
shouldn't be - the host names are in the hosts file
2015-05-28 23:52:25
mgriffin
jafa: did you check for ist vs sst?
2015-05-28 23:58:29
jafa
what is ist?
2015-05-28 23:58:41
jafa
sst method configured to use rsync
2015-05-29 00:00:08
mgriffin
http://galeracluster.com/documentation-webpages/statetransfer.html
2015-05-29 00:01:04
jafa
ok
2015-05-29 00:03:05
jafa
what should I be checking RE ist vs sst?
2015-05-29 00:04:30
mgriffin
you seemed confused about why start up might be very fast vs slow earlier
2015-05-29 00:04:34
mgriffin
this is a reason
2015-05-29 00:05:22
jafa
ok, so it is normal for startup to sometimes take a minute or two? (I am fine with that if normal - will work on tracking down the timeout)
2015-05-29 00:48:57
mgriffin
seems like maybe the init script should read MYSQLD_STARTUP_TIMEOUT from /etc/default but does not
2015-05-29 01:44:06
nvidhive
jafa: are you warming up a large buffer pool?
2015-05-29 01:47:31
nvidhive
errr "innodb-buffer-pool-populate"
2015-05-29 01:48:27
nvidhive
that one was making my startup times quite long
2015-05-29 02:30:54
wcooke
Hi All!
2015-05-29 02:32:25
mgriffin
nvidhive: that variable should increase startup time but does not "warm up" the buffer pool, it allocates it
2015-05-29 02:32:35
mgriffin
nvidhive: you might be thinking of innodb_blocking_buffer_pool_restore
2015-05-29 02:32:43
nvidhive
right
2015-05-29 02:32:49
nvidhive
that is why I corrected myself
2015-05-29 02:32:57
nvidhive
"errr "
2015-05-29 02:33:13
mgriffin
ah